Streamline the memcpy a bit removing some unnecessary instructions.
The biggest speed improvement comes from changing the size of
the preload. On krait, the sweet spot for the preload in the main
loop is twice the L1 cache line size.
In most cases, these small tweaks yield > 1000MB/s speed ups. As
the size of the memcpy approaches about 1MB, the speed improvement
disappears.
Change-Id: Ief79694d65324e2db41bee4707dae19b8c24be62
This uses the new code original submitted as memcpy.a15.S as
the base. However, the old code handled unaligned src/dst better
so that was spliced in. I optimized the original unaligned code by
removing a few unnecessary instructions. I optimized the a15 code by
rewriting the pre and post code. I also modified the main loop to add
a pld so that larger copies would not stall waiting for memory.
Test cases for the new memcpy:
- Copy all sized values from 0 to 1024 bytes, using whatever alignment
is returned by malloc.
For each alignment case described below, the test copied from 0 to 128
bytes.
- Src and dst pointers are both aligned to the same value, starting
at one going through every power of two up to and including 128.
- Src aligned to double word boundary, dst aligned to word boundary.
- Src aligned to word boundary, dst aligned to double word boundary.
- Src aligned to 16 bit boundary, dst aligned to word boundary.
- Src aligned to word boundary, dst aligned to 16 byte boundary.
- Src aligned to word boundary, dst aligned to 1 byte from a word
boundary.
- Src aligned to word boundary, dst aligned to 2 bytes from a word
boundary.
- Src aligned to word boundary, dst aligned to 3 bytes from a word
boundary.
- Src aligned to 1 byte from a word boundary, dst aligned to a word
boundary.
- Src aligned to 2 bytes from a word boundary, dst aligned to a word
boundary.
- Src aligned to 3 bytes from a word boundary, dst aligned to a word
boundary.
Cases to verify the unaligned source code properly aligns to a 16 bit
boundary.
- Src aligned to 1 byte from a 128 bit boundary, dst aligned to
4 + 128 bit boundary.
- Src aligned to 1 byte from a 128 bit boundary, dst aligned to
8 + 128 bit boundary.
- Src aligned to 1 byte from a 128 bit boundary, dst aligned to
12 + 128 bit boundary.
- Src aligned to 1 byte from a 128 bit boundary, dst aligned to
16 + 128 bit boundary.
In all cases, a two byte fencepost was placed at the end of the
destination to verify that only the requested number of bytes were copied.
Bug: 8005082
Merge from internal master.
(cherry-picked from commit 21ede92d79)
Change-Id: Ief70c9e6dc8c6473ae245b6570b2c266fed9618c
This lets us move all the ARM syscall stubs over to the kernel <asm/unistd.h>.
Our generated <sys/linux-syscalls.h> is now unused, but I'll remove that in a
later change.
Change-Id: Ie5ff2cc4abce1938576af7cbaef615a79c7f310d
For some reason, socketcalls.c was only being compiled for ARM, where
it makes no sense. For x86 we generate stubs for the socket functions
that use __NR_socketcall directly.
Change-Id: I84181e6183fae2314ae3ed862276eba82ad21e8e
<sys/linux-syscalls.h> only contains constants for the syscalls
we're generating stubs for. We want all the syscalls available
on the architecture in question.
Keep using <sys/linux-syscalls.h> on ARM for now because the
__NR_ARM_set_tls and __NR_ARM_cacheflush values aren't in <asm/unistd.h>.
Change-Id: I66683950d87d9b18d6107d0acc0ed238a4496f44
This uses the new strcmp.a15.S code as the basis for new versions
of strcmp.S.
The cortex-a15 code is the performance optimized version of strcmp.a15.S
taken with only the addition of a few pld instructions.
The cortex-a9 code is the same as the cortex-a15 code except that the
unaligned strcmp code was taken from the original strcmp.S.
The krait code is the same as the cortex-a15 code except that one path
in the unaligned strcmp code was taken from the original strcmp.S code
(the 2 byte overlap case).
The generic code is the original unmodified strmp.S from the bionic
subdirectory.
All three new versions underwent these test cases:
Strings the same, all same size:
- Both pointers double word aligned.
- One pointer double word aligned, one pointer word aligned.
- Both pointers word aligned.
- One pointer double word aligned, one pointer 1 off a word alignment.
- One pointer double word aligned, one pointer 2 off a word alignment.
- One pointer double word aligned, one pointer 3 off a word alignment.
- One pointer word aligned, one pointer 1 off a word alignment.
- One pointer word aligned, one pointer 2 off a word alignment.
- One pointer word aligned, one pointer 3 off a word alignment.
For all cases where it made sense, the two pointers were also tested
swapped.
Different strings, all same size:
- Single difference at double word boundary.
- Single difference at word boudary.
- Single difference at 1 off a word alignment.
- Single difference at 2 off a word alignment.
- Single difference at 3 off a word alignment.
Different sized strings, strings the same until the end:
- Shorter string ends on a double word boundary.
- Shorter string ends on word boundary.
- Shorter string ends at 1 off a word boundary.
- Shorter string ends at 2 off a word boundary.
- Shorter string ends at 3 off a word boundary.
For all different cases, run them through the same pointer alignment
cases when the strings are the same size.
For all cases the two pointers were also tested swapped.
Bug: 8005082
Merge from internal master.
(cherry-picked from commit a9a5870d16)
Change-Id: I4c2b98f8a50804fb98ab67f75e9d660f1315a144
We only need one logging API, and I prefer the one that does no
allocation and is thus safe to use in any context.
Also use O_CLOEXEC when opening the /dev/log files.
Move everything logging-related into one header file.
Change-Id: Ic1e3ea8e9b910dc29df351bff6c0aa4db26fbb58
Move arch specific code for arm, mips, x86 into separate
makefiles.
In addition, add different arm cpu versions of memcpy/memset.
Bug: 8005082
Merge from internal master (acdde8c1cf).
Change-Id: I04f3d0715104fab618e1abf7cf8f7eec9bec79df
The attached patch provides a new implementation of strcmp for ARM,
using LDRD instead of LDR whenever possible.
For older architectures that do not support LDRD, this implementation
uses the same algorithm as before.
Testing and benchmarking:
* Validation: successfully passes a test that compares different strings
of length 1-128 and offsets 0-8 from a word boundary. Checked on
qemu/A15/A9, ARM/Thumb mode, Big/Little Endian.
* Integration with gcc: no regression on qemu for arm-none-eabi --with-cpu
a15/a9 --with-mode arm/thumb.
Change-Id: I9e230e1b99dbdc9119b69ee858a89038c516a4ea
Signed-off-by: Vassilis Laganakos <vasileios.laganakos@arm.com>
The strategy for large block sizes is LDRD and STRD with offset addressing,
where the main loop copies 64 bytes in every iteration, (i.e., 8 calls to
LDRD and STRD pairs), interleaving load and stores (i.e., the pairs of LDRD
and STRD of the same data are consecutive instructions), and the writeback
of an updated address is a separate instruction, which allows us to write
back the accumulated update once per iteration.
This strategy is implemented in memcpy.S. In some configurations, a plain
version of memcpy (included from memcpy-stub.c) is used instead of the
optimized one.
Validation:
* Correctness: checked memcpy using a test harness for block sizes
ranging between 1 to 128, and source and destination buffers alignment
ranging in { 0,1,2,3,4,8,12 } bytes each.
* Performance: benchmarking on Cortex-A15 FPGA indicates that this strategy
is better for A15 than the strategy used by glibc and even slightly better
than using NEON. Benchmarking on Cortex-A9 bare metal and Linux shows
that the proposed strategy is reasonable: not as fast as the version of
memcpy from glibc (which is the best open source strategy for A9), but
comparable with csl and bionic.
* Integration with GCC: no regression for arm-none-eabi --with-cpu
cortex-a15 and cortex-a9.
Change-Id: Ied56354d8992c62ae3e02d582a2bd55585d814b9
Signed-off-by: Vassilis Laganakos <vasileios.laganakos@arm.com>
Fix the pthread_setname_np test to take into account that emulator kernels are
so old that they don't support setting the name of other threads.
The CLONE_DETACHED thread is obsolete since 2.5 kernels.
Rename kernel_id to tid.
Fix the signature of __pthread_clone.
Clean up the clone and pthread_setname_np implementations slightly.
Change-Id: I16c2ff8845b67530544bbda9aa6618058603066d
If r0 == 0, we're the child. If r0 > 0, we're the parent.
Otherwise set errno.
The __bionic_clone code I copy & pasted was wrong. This patch
fixes both.
Bug: 3461078
Change-Id: Ibb7d6cc7e54e666841f2f0dc59a141a0b31982e4
MIPS and x86 appear to have been correct already.
(Also fix unit tests that ASSERT_EQ with errno so that the
arguments are in the retarded junit order.)
Bug: 3461078
Change-Id: I2418ea98927b56e15b4ba9cfec97f5e7094c6291
There's now only one place where we deal with this stuff, it only needs to
be parsed once by the dynamic linker (rather than by each recipient), and it's
now easier for us to get hold of auxv data early on.
Change-Id: I6314224257c736547aac2e2a650e66f2ea53bef5
This brings us up to date with FreeBSD HEAD, fixes various bugs, unifies
the set of functions we support on ARM, MIPS, and x86, fixes "long double",
adds ISO C99 support, and adds basic unit tests.
It turns out that our "long double" functions have always been broken
for non-normal numbers. This patch fixes that by not using the upstream
implementations and just forwarding to the regular "double" implementation
instead (since "long double" on Android is just "double" anyway, which is
what BSD doesn't support).
All the tests pass on ARM, MIPS, and x86, plus glibc on x86-64.
Bug: 3169850
Bug: 8012787
Bug: https://code.google.com/p/android/issues/detail?id=6697
Change-Id: If0c343030959c24bfc50d4d21c9530052c581837
Replace a kernel header file dependency with files from NetBSD.
They're more complete, and ELF is ELF, whether you're on Linux or a BSD.
Bug: 7973611
Change-Id: I83ee719e7efdf432ec2ddbe8be271d05b2f558d7
We had two copies of the backtrace code, and two copies of the
libcorkscrew /proc/pid/maps code. This patch gets us down to one.
We also had hacks so we could log in the malloc debugging code.
This patch pulls the non-allocating "printf" code out of the
dynamic linker so everyone can share.
This patch also makes the leak diagnostics easier to read, and
makes it possible to paste them directly into the 'stack' tool (by
using relative PCs).
This patch also fixes the stdio standard stream leak that was
causing a leak warning every time tf_daemon ran.
Bug: 7291287
Change-Id: I66e4083ac2c5606c8d2737cb45c8ac8a32c7cfe8
In the default case, we don't need to use the stack, we can save r7 with
ip register (that what does eglibc).
This allow to fix vfork data corruption
(see 3884bfe966), because vfork now don't
use the stack.
If the platform code is compiled with -mcpu=cortex-a15, then without this
change prebuilt libraries built against -march=armv7 cannot resolve the
dependency on __aeabi_idiv (provided by libgcc.a).
Bug: 7961327
cherry-picked from internal master.
Change-Id: I8fe59a98eb53d641518b882523c1d6a724fb7e55
Add signalfd() call to bionic.
Adding the signalfd call was done in 3 steps:
- add signalfd4 system call (function name and syscall
number) to libc/SYSCALLS.TXT
- generate all necessary headers by calling
libc/tools/gensyscalls.py. This patch is adding
the generated files since the build system
does not call gensyscalls.py.
- create the signalfd wrapper in signalfd.cpp and add
the function prototype to sys/signalfd.h
(cherry-pick of 0c11611c11, modified to
work with older versions of GCC still in use on some branches.)
Change-Id: I4c6c3f12199559af8be63f93a5336851b7e63355
Previously we'd been relying on getting the machine-specific <endian.h>
instead of the top-level <endian.h>, and <sys/endian.h> was basically broken.
Now, with this patch and the previous patch we should have <endian.h>
and <sys/endian.h> behaving the same. This is basically how NetBSD's endian.h
works, and was probably how ours was originally intended to work.
Bug: http://code.google.com/p/android/issues/detail?id=39824
Change-Id: I71de5a507e633de166013a658b5764df9e1aa09c
The near duplicates upset fussier compilers that insist that
typedefs be exactly the same, but the fix isn't to make all
copies identical...
Change-Id: Icfdace41726f36ec33c9ae919dbb5a54d3529cc9
We'd manually hacked _BYTE_ORDER into the arm and mips "_types.h" headers,
but not into the x86 one. Judging by upstream, _BYTE_ORDER should be in
the "endian.h" headers instead, so let's uniformly do that.
I've also ironed out some of the other differences between the different
architectures' header files too.
Bug: http://code.google.com/p/android/issues/detail?id=39824
Change-Id: I19d3af7ffd74e1c02b1b6886aec0f0d11f44ab8d
Adds new code to function memset, optimized for Cortex A9.
Copyright (C) ST-Ericsson SA 2010
Added neon implementation
Author: Henrik Smiding henrik.smiding@stericsson.com for ST-Ericsson.
Change-Id: Id3c87767953439269040e15bd30a27aba709aef6
Signed-off-by: Christian Bejram <christian.bejram@stericsson.com>
Adds new code to memcpy function, optimized for Cortex A9.
Adds new ARM-only loop, for operations where source and
destination are aligned.
Copyright (C) ST-Ericsson SA 2010
Modified neon implementation to fit Cortex A9 cache line size,
for those running 32 bytes L2 cache line size.
Also split the implementation in aligned and unaligned access,
for those that allows unaligned memory access with Neon.
For totally aligned operations, arm-only code is used.
Change-Id: I95ebf6164cd6486b12a7e3e98e369db21e7e18d2
Author: Henrik Smiding henrik.smiding@stericsson.com for ST-Ericsson.
Signed-off-by: Christian Bejram <christian.bejram@stericsson.com>
This patch updates the C library headers to provide ucontext_t
definitions for three architectures.
+ Fix <signal.h> to always define 'struct sigcontext'.
The new declarations are announced with new macros defined in
<sys/cdefs.h> in order to make it easier to adapt client code
that already defines its own, incompatible, versions of the
structures seen here.
http://code.google.com/p/android/issues/detail?id=34784
Change-Id: Ie78c48690a4ce61c50593f6c39639be7fead3596
Add getsid() system call to bionic for
all architectures. This is needed for various tools
(e.g. perf).
Adding the getsid system call was done in 3 steps:
() add getsid system call (function name and syscall
number) to libc/SYSCALLS.TXT
() generate all necessary headers by calling
libc/tools/gensyscalls.py. This patch is adding
the generated files since the build system
does not call gensyscalls.py.
() add the system call signature to libc/include/unistd.h
Change-Id: Id69a257e13ec02e1a44085a6b217a3f19ab025b1
Signed-off-by: Irina Tirdea <irina.tirdea@intel.com>
Legacy ARM shared libraries use this generic version of atexit(),
which queues exit functions for invocation at program exit, at
which time the library may have been dlclose()'d, causing the
program to crash.
Change-Id: I41ae153c23268daa65ede7fb8966fc3e9caec369
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@gmail.com>
To properly support legacy ARM shared libraries, libc.so needs
to export the symbols __dso_handle and atexit, even though
these are now supplied by the crt startup code.
This patch reshuffles the existing CRT_LEGACY_WORKAROUND
conditionally compiled code slightly so it works as the
original author likely intended.
Change-Id: Id6c0e94dc65b7928324a5f0bad7eba6eb2f464b9
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@gmail.com>
The runtime linker parses the ELF section headers to
discover the size of the init_array and fini_array, so
there is no point in putting NULL terminators at the end.
Change-Id: I3246cd585efce9314155600277dd829e9f37d04f
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@gmail.com>
None of the supported ARCHs actually populate these sections,
so there is no point in keeping them in the binaries.
Change-Id: I21a364f510118ac1114e1b49c53ec8c895c6bc6b
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@gmail.com>
Useful if you're trying to defeat ASLR, otherwise not
so much ...
Change-Id: I17ebb50bb490a3967db9c3038f049adafe2b8ea7
Signed-off-by: Ard Biesheuvel <ard.biesheuvel@gmail.com>
This header is used on bionic build and should be propagated into
sysroot on toolchain rebuild. Discussion re. this header is here:
http://gcc.gnu.org/ml/gcc-patches/2012-08/msg00936.html
It is available already in mips NDK platforms:
development/ndk/platforms/android-9/arch-mips/include/link.h
Change-Id: I39ff467cdac9f448e31c11ee3e14a6200e82ab57
Signed-off-by: Pavel Chupin <pavel.v.chupin@intel.com>
Add a GNU_STACK marker to crtend* files. This tells the linker
that these files do not require an executable stack.
When linking, a missing GNU_STACK marker in any .o file can prevent
the compiler from automatically marking the final executable as NX
safe (executable stack not required). In Android, we normally work
around this by adding -Wa,--noexecstack / -Wl,-z,noexecstack.
For files like crtend.S / crtend_so.S, which are included in every
executable / shared library, it's better to add the GNU_STACK note
directly to the assembly file. This allows the compiler to
automatically mark the final executable as NX safe without any
special command line options.
References: http://www.gentoo.org/proj/en/hardened/gnu-stack.xml
Change-Id: I07bd058f9f60ddd8b146e0fb36ba26ff84c0357d
Move the stackpointer so a captured signal does not corrupt
stack variables needed for __thread_entry.
Change-Id: I3e1e7b94a6d7cd3a07081f849043262743aa8064
Rewrite
crtbegin.S -> crtbegin.c
crtbegin_so.S -> crtbegin_so.c
This change allows us to generate PIC code without relying
on text relocations.
As a consequence of this rewrite, also rewrite
__dso_handle.S -> __dso_handle.c
__dso_handle_so.S -> __dso_handle_so.c
atexit.S -> atexit.c
In crtbegin.c _start, place the __PREINIT_ARRAY__, __INIT_ARRAY__,
__FINI_ARRAY__, and __CTOR_LIST__ variables onto the stack, instead of
passing a pointer to the text section of the binary.
This change appears sorta wonky, as I attempted to preserve,
as much as possible, the structure of the original assembly.
As a result, you have C files including other C files, and other
programming uglyness.
Result: This change reduces the number of files with text-relocations
from 315 to 19 on my Android build.
Before:
$ scanelf -aR $OUT/system | grep TEXTREL | wc -l
315
After:
$ scanelf -aR $OUT/system | grep TEXTREL | wc -l
19
Change-Id: Ib9f98107c0eeabcb606e1ddc7ed7fc4eba01c9c4
crtbegin_dynamic and crtbegin_static are essentially identical,
minus a few trivial differences (comments and whitespace).
Eliminate duplicates.
Change-Id: Ic9fae6bc9695004974493b53bfc07cd3bb904480
Adds new code to function memcmp, optimized for Cortex A9.
Copyright (C) ST-Ericsson SA 2010
Added neon optimization
Change-Id: I8864d277042db40778b33232feddd90a02a27fb0
Author: Henrik Smiding henrik.smiding@stericsson.com for ST-Ericsson.
Signed-off-by: Christian Bejram <christian.bejram@stericsson.com>
Modify the dynamic linker so that executables can be loaded
at locations other than 0x00000000.
Modify crtbegin* so that non-PIC compilant "thumb interwork
veneers" are not created by the linker.
Bug: 5323301
Change-Id: Iece0272e2b708c79034f302c20160e1fe9029588
Some SoCs that support NEON nevertheless perform better with a non-NEON than a
NEON memcpy(). This patch adds build variable ARCH_ARM_USE_NON_NEON_MEMCPY,
which can be set in BoardConfig.mk. When ARCH_ARM_USE_NON_NEON_MEMCPY is
defined, we compile in the non-NEON optimized memcpy() even if the SoC supports
NEON.
Change-Id: Ia0e5bee6bad5880ffc5ff8f34a1382d567546cf9
So that we can always get the full stack trace regardless of gcc's handling
of the "noreturn" attribute associated with abort().
(Cherry pick of Id264a5167e7cabbf11515fbc48f5469c527e34d4.)
Bug: 6455193
Conflicts:
libc/Android.mk
Change-Id: I568fc5303fd1d747075ca933355f914122f94dac
So that we can always get the full stack trace regardless of gcc's handling
of the "noreturn" attribute associated with abort().
[cherry-picked from master]
BUG:6455193
Change-Id: I0102355f5bf20e636d3feab9d1424495f38e39e2
ARM Cortex A8 use 64 bytes and ARM Cortex A9 use 32 bytes cache line
size.
The following patch:
Adds code to adjust memcpy cache line size to match A9 cache line
size.
Adds a flag to select between 32 bytes and 64 bytes cache line
size.
Copyright (C) ST-Ericsson SA 2010
Modified neon implementation to fit Cortex A9 cache line size
Author: Henrik Smiding henrik.smiding@stericsson.com for
ST-Ericsson.
Change-Id: I8a55946bfb074e6ec0a14805ed65f73fcd0984a3
Signed-off-by: Christian Bejram <christian.bejram@stericsson.com>
The END macro was put too far down which made the linker complain about
it. Move up to the end of the code.
Change-Id: Ica71a9c6083b437d2213c7cefe34b0083c78f16b
Marking segments read-only was pushing the alignment of __on_dlclose by
2 bytes making it unaligned. This change makes sure the ARM code is
aligned to the 4 byte boundary.
Bug: 6313309
Change-Id: Ic2bf475e120dd61225ec19e5d8a9a8b1d0b7f081
This change fixes a segmentation fault in the libc unwinder when it goes
past __libc_init.
Unwind instructions for __libc_init direct it to grab the return address from
the stack frame. Without this change, the unwinder gets a wild address and
looks up further unwind instructions for the routine at that address. If it's
unlucky enough to hit an existing function, it will try to unwind it. Bad
things happen then.
With this change, the return address always points to the _start function,
which does not have unwind instructions associated with it. This stop the
unwind process.
__libc_init never returns, so this does not affect program execution, other
than adding 4 bytes on the main thread stack.
Change-Id: Id58612172e8825c8729cccd081541a13bff96bd0
Use the same pattern in atexit.S to reference __dso_handle in a way that
doesn't require a TEXTREL flag to be set.
Change-Id: Id69d20863ee203d2b2f7ef0db230f9b548657741
Some platform libraries built for ICS do not work with master
because of some refactoring in frameworks/base.
Make sure that these libgcc symbols are always present in our libc
Change-Id: Ib8d345878be0ba711f051082a778f5cc1f1b3a19
Signed-off-by: Dima Zavin <dima@android.com>
This patch defines a few new macros that can be used to control the
visibility of symbols exported by the C library:
- ENTRY_PRIVATE() can be used in assembly sources to indicate
that an assembler function should have "hidden" visibility, i.e.
will never be exported by the C library's shared library.
This is the equivalent of using __LIBC_HIDDEN__ for a C function,
but ENTRY_PRIVATE() works like ENTRY(), and must be used with
END() to tag the end of the function.
- __LIBC_ABI_PUBLIC__ can be used to tag a C functions as being
part of the C library's public ABI. This is important for a
few functions that must be exposed by the NDK to maintain
binary compatibility.
Once a symbol has been tagged with this macro, it shall
*never* be removed from the library, even if it becomes
directly unused due to implementation changes
(e.g. __is_threaded).
- __LIBC_ABI_PRIVATE__ can be used for C functions that should
always be exported by the C library because they are used by
other libraries in the platform, but should not be exposed
by the NDK. It is possible to remove such symbols from the
implementation if all callers are also modified.
+ Add missing END() assembly macro for x86
Change-Id: Ia96236ea0dbec41d57bea634b39d246b30e5e234
The xattr system calls are required for the SE Android userspace in
order to get and set file security contexts. In particular, libselinux
requires these calls.
Change-Id: I78f5eb3d8f3384aed0a5e7c6a6f001781d982017
