This commit adds a simple check to see if the offset of the read
request matches the expected file offset.
We could try to recover, from this condition at some point in the future.
Right now it is better to return an error instead of corrupted data.
This commit ensures that we have sent at least one read request before
we try to read data in sftp_read().
Otherwise sftp_read() would return 0 bytes (indicating EOF) if the
socket is not ready for writing.
Since we can only store data from a single chunk in filep,
we have to stop receiving data as soon as the buffer is full.
This adresses the following bug report:
https://github.com/libssh2/libssh2/issues/50
"agent_disconnect_unix", called by "libssh2_agent_disconnect", was
leaving the file descriptor in the agent structure unchanged. Later,
"libssh2_agent_free" would call again "libssh2_agent_disconnect" under
the hood and it would try to close again the same file descriptor. In
most cases that resulted in just a harmless error, but it is also
possible that the file descriptor had been reused between the two
calls resulting in the closing of an unrelated file descriptor.
This patch sets agent->fd to LIBSSH2_INVALID_SOCKET avoiding that
issue.
Signed-off-by: Salvador Fandiño <sfandino@yahoo.com>
The Qc3 library is not able to handle PKCS#8 EncryptedPrivateKeyInfo structures
by itself. It is only capable of decrypting the (encrypted) PrivateKeyInfo
part, providing a key encryption key and an encryption algorithm are given.
Since the encryption key and algorithm description part in a PKCS#8
EncryptedPrivateKeyInfo is a PKCS#5 structure, such a decoder is needed to
get the derived key method and hash, as well as encryption algorith and
initialisation vector.
The Qc3 library requires a minimum key length depending on the target
hash algorithm. Append binary zeroes to the given key if not long enough.
This matches RFC 2104 specifications.
The Qc3 library requires the key encryption key to exist as long as
the encrypted key is used. Its descriptor token is then kept as an
"encrypted key slave" for recursive release.
Build procedure extproto() did not strip braces from header files, thus
possibly prepended them to true prototypes. This prevented the prototype to
be recognized as such.
The solution implemented here is to map braces to semicolons, effectively
considering them as potential prototype delimiters.
In addition, file os400/macros.h declares all procedures originally
defined as macros. It must not be used for real inclusion and is only
intended to be used as a `database' for macro wrapping procedures generation.
Some structure fields holding callback addresses have the same name as the
underlying system function (connect, send, recv). Set parentheses around
their reference to suppress a possible macro substitution.
Use a macro for connect() on OS/400 to resolve a const/nonconst parameter
problem.
OS/400 crypto library is unable to sign a precomputed SHA1 hash: however
it does support a procedure that hashes data fragments and rsa signs.
If defined, the new macro _libssh2_rsa_sha1_signv() implements this function
and disables use of _libssh2_rsa_sha1_sign().
The function described above requires that the struct iovec unused slacks are
cleared: for this reason, macro libssh2_prepare_iovec() has been introduced.
It should be defined as empty for crypto backends that are not sensitive
to struct iovec unused slack values.
* close https://github.com/libssh2/libssh2/issues/69
* sync a declaration with the rest of similar ones
* handle EVP_MD_CTX_new() returning NULL with OpenSSL 1.1.0
* fix potential memory leak with OpenSSL 1.1.0 in
_libssh2_*_init() functions, when EVP_MD_CTX_new() succeeds,
but EVP_DigestInit() fails.
_libssh2_wincng_hash_final was returning the internal BCRYPT
status code instead of a valid libssh2 return value (0 or -1).
This also means that _libssh2_wincng_hash never returned 0.
Net::SSH2, the Perl wrapping module for libssh2 implements several features*
on top of libssh2 that can fail and so need some mechanism to report the error
condition to the user.
Until now, besides the error state maintained internally by libssh2, another
error state was maintained at the Perl level for every session object and then
additional logic was used to merge both error states. That is a maintenance
nighmare, and actually there is no way to do it correctly and consistently.
In order to allow the high level language to add new features to the library
but still rely in its error reporting features the new function
libssh2_session_set_last_error (that just exposses _libssh2_error_flags) is
introduced.
*) For instance, connecting to a remote SSH service giving the hostname and
port.
Signed-off-by: Salvador Fandino <sfandino@yahoo.com>
Signed-off-by: Salvador Fandiño <sfandino@yahoo.com>
Before this patch "_libssh2_error" required the error message to be a
static string.
This patch adds a new function "_libssh2_error_flags" accepting an
additional "flags" argument and specifically the flag
"LIBSSH2_ERR_FLAG_DUP" indicating that the passed string must be
duplicated into the heap.
Then, the method "_libssh2_error" has been rewritten to use that new
function under the hood.
Signed-off-by: Salvador Fandino <sfandino@yahoo.com>
Signed-off-by: Salvador Fandiño <sfandino@yahoo.com>
After reading the public key from file the size was incorrectly
decremented by one.
This was usually a harmless error as the last character on the public
key file is an unimportant EOL. But if due to some error the public key
file is empty, the public key size becomes (uint)(0 - 1), resulting in
an unrecoverable out of memory error later.
Signed-off-by: Salvador Fandi??o <sfandino-/E1597aS9LQAvxtiuMwx3w@public.gmane.org>
A common novice programmer error (at least among those using the
wrapping Perl module Net::SSH2), is to try to reuse channels.
This patchs detects that incorrect usage and fails with a
LIBSSH2_ERROR_BAD_USE error instead of hanging.
Signed-off-by: Salvador Fandino <sfandino-/E1597aS9LQAvxtiuMwx3w@public.gmane.org>
Implement support for these algorithms and wire them up to the libgcrypt
and OpenSSL backends. Increase the maximum MAC buffer size to 64 bytes
to prevent buffer overflows. Prefer HMAC-SHA-256 over HMAC-SHA-512, and
that over HMAC-SHA-1, as OpenSSH does.
Closes#40
libssh2 equivalent of curl patch d21b66835f
This allows to build for the non-default target when using a multi-target mingw distro.
Also bump default OpenSSL dependency path to 1.0.2c.
VS2015 moved stdio functions to the header files as inline function. That means check_function_exists can't detect them because it doesn't use header files - just does a link check. Instead we need to use check_symbol_exists with the correct headers.
Despite we announced the CMake support in libssh2-1.6.0 release notes,
the files required by the CMake build system were not included in the
release tarballs. Hence, the only way to use CMake for build was the
upstream git repository.
This commit makes CMake actually supported in the release tarballs.
Do not create symbolic links off the build directory. Recent autotools
verify that out-of-source build works even if the source directory tree
is not writable.
The function sftp_read never return more then 2000 bytes (as it should
when I asked Daniel). I increased the MAX_SFTP_READ_SIZE to 30000 but
didn't get the same speed as a sftp read in SecureSSH. I analyzed the
code and found that a return always was dona when a chunk has been read.
I changed it to a sliding buffer and worked on all available chunks. I
got an increase in speed and non of the test I have done has failed
(both local net and over Internet). Please review and test. I think
30000 is still not the optimal MAX_SFTP_READ_SIZE, my next goal is to
make an API to enable changing this value (The SecureSSH sftp_read has
more complete filled packages when comparing the network traffic)
The error message returned by libssh2_channel_open in case of a server side channel open failure is now more detailed and includes the four standard error conditions in RFC 4254.
This reverts commit 2d2744efdd0497b72b3e1ff6e732aa4c0037fc43.
Autobuilds show that this did not solve the issue.
And it seems like RtlFillMemory is defined to memset,
which would be optimized out by some compilers.
This takes 22bd8d81d8fab956085e2079bf8c29872455ce59 and
b8289b625e291bbb785ed4add31f4759241067f3 into account,
but still makes it enabled by default if it is supported
and error out in case it is unsupported and was requested.
Reduced number of calls to strlen, because shell_quotearg already
returns the length of the resulting string (e.q. quoted path)
which we can add to the existing and known cmd_len.
Removed obsolete call to memset again, because we can put a final
NULL-byte at the end of the string using the calculated length.
This re-introduces the original feature proposed during
the development of the WinCNG crypto backend. It still needs
to be added to libssh2 itself and probably other backends.
Memory is cleared using the function SecureZeroMemory which is
available on Windows systems, just like the WinCNG backend.
The new feature isn't implemented for the WinCNG backend currently, but the WinCNG backend didn't contain any implementation of the required backend functions - even ones that returns an error. That caused link errors.
This change fixes the problem by providing an implementation of the backend functions that returns an error.
Autotools expects a configuration template file at src/libssh2_config.h.in, which buildconf generates. But the CMake build system has its CMake-specific version of the file at this path. This means that, if you don't run buildconf, the Autotools build will fail because it configured the wrong header template.
See https://github.com/libssh2/libssh2/pull/8.
The sshd test fixture was returning -1 if an error occurred, but negative error codes aren't technically valid (google it). Bash on Windows converted them to 0 which made setup failure look as though all tests were passing.
Allow to pass custom `CFLAGS` options via environment variable
`LIBSSH2_CFLAG_EXTRAS`. Default and automatically added options of
`GNUmakefile` have preference over custom ones. This addition is useful
for passing f.e. custom CPU tuning or LTO optimization (`-flto
-ffat-lto-objects`) options. The only current way to do this is to edit
`GNUmakefile`. This patch makes it unnecessary.
This is a mirror of similar libcurl patch:
https://github.com/bagder/curl/pull/136
... if the pointer subtraction of sp1 - pubkey - 1 resulted in a
negative or larger value than pubkey_len, memchr would fail.
Reported by Coverity CID 89846.
... host cannot be NULL in line 525, because it is always
valid (e.g. at least set to "0.0.0.0") after lines 430 and 431.
Reported by Coverity CID 89807.
Fixes VS2012 code analysis warning C6386:
buffer overrun: accessing 'pbOutput', the writable size is
'cbOutput' bytes, but '3' bytes may be written: libssh2 wincng.c 610
Fixes VS2012 code analysis warning C6387: 'p+4' may be '0':
this does not adhere to the specification for the function
'memcpy': libssh2 agent.c 330
Fixes VS2012 code analysis warning C6387: 'p' may be '0':
this does not adhere to the specification for the function
'UnmapViewOfFile': libssh2 agent.c 333
test/GNUmakefile: added architecture autodetection; added switches to
CFLAGS and RCFLAGS to make sure that the right architecture is used.
Added support to build with WinCNG.
Added architecture autodetection; added switches to CFLAGS and
RCFLAGS to make sure that the right architecture is used.
Added support to build with WinCNG.
Initially reported by Bob Kast as "for MS VS builds, specify the
libraries that are required so they don't need to go into all
project files that may use this library". Thanks a lot.
If you are building a DLL, then you need to explicitly export each
entry point. When building a static library, you should not.
libssh2 was exporting the entry points whether it was building a DLL or a
static library. To elaborate further, if libssh2 was used as a static
library, which was being linked into a DLL, the libssh2 API would be
exported from that separate DLL.
This avoids line-wrapping in between parameters and makes the
error message look like the following:
configure: error: No crypto library found!
Try --with-libssl-prefix=PATH
or --with-libgcrypt-prefix=PATH
or --with-wincng on Windows
Initially reported by Bob Kast as "Wincng - define function
prototypes for wincng routines". Thanks a lot.
Also replaced structure definitions with type definitions.
Inspired by Bob Kast's reports, this commit enables the compilation
of libssh2 with WinCNG using the generated Visual Studio project files.
This commit adds WinCNG support to parts of the existing Win32 build
infrastructure, until new build systems, like pre-defined VS project
files or CMake files may be added.
This commit and b20bfeb3e519119a48509a1099c06d65aa7da1d7 raise one
question: How to handle build systems, like VS project files, that
need to include all source files regardless of the desired target,
including all supported crypto backends? For now the mentioned commit
added a check for LIBSSH2_OPENSSL to openssl.c and with this commit
the supported crypto backends are hardcoded within Makefile.am.
Removed header file combination that is not supported on a real
Windows platform and can only be compiled using MinGW. Replaced
custom NTSTATUS return code checks with BCRYPT_SUCCESS macro.
../src/knownhost.c: In function 'libssh2_knownhost_readline':
../src/knownhost.c:651:16: warning: 'key_type_len' may be used
uninitialized in this function [-Wmaybe-uninitialized]
rc = knownhost_add(hosts, hostbuf, NULL,
^
../src/knownhost.c:745:12: note: 'key_type_len' was declared here
size_t key_type_len;
^
Commit d512b25f69a1b6778881f6b4b5ff9cfc6023be42 introduced a crypto
library abstraction in the autotools build system, to allow us to more
easily support new crypto libraries. In that process it was found that
all other build system which we support are hard-coded to build with
OpenSSL. Commit f5c1a0d98bd51aeb24aca3d49c7c81dcf8bd858d fixes automake
introduced into non-autotools build systems but still overlooked the
CPP macro saying that we are using OpenSSL.
Thanks to Marc Hörsken for identifying this issue and proposing a fix
for win32/{GNUmakefile,config.mk}. This commit uses a slightly different
approach but the end result is the same.
Commit 85c6627c changed the behaviour of `libssh2_knownhost_writeline` so that it stopped returning the number of bytes needed when the given buffer was too small. Also, the function changed such that is might write to part of the buffer before realising it is too small.
This commit restores the original behaviour, whilst keeping the unknown-key-type functionality that 85c6627c. Instead of writing to the buffer piecemeal, the length of the various parts is calculated up front and the buffer written only if there is enough space. The calculated necessary size is output in `outlen` regardless of whether the buffer was written to.
The main use-case for the original behaviour that this commit restores is to allow passing in a NULL buffer to get the actual buffer size needed, before calling the function again with the buffer allocated to the exact size required.
The first might occur if _libssh2_packet_add returns an error, as
fullpacket_state wasn't reset to idle so if it were possible for
fullpacket to be called again, it would return to the same state
handler and re-use the freed p->packet buffer.
The second could occur if decrypt returned an error, as it freed the
packet buffer but did not clear total_num, meaning that freed buffer
could be written into again later.
In one case, the error code from `_libssh2_transport_read` was being returned from `_libssh2_channel_write` without setting it as the last error by calling `_libssh2_error`. This commit fixes that.
Found when using a session whose socket had been inadvertently destroyed. The calling code got confused because via `libssh2_session_last_error` it appeared no error had occurred, despite one being returned from the previous function.
When using the OpenSSL libraries in FIPS mode, the function call
EVP_DigestInit() is actually #defined to FIPS_digestinit().
Unfortunately wheres EVP_DigestInit() initialises the context and then
calls EVP_DigestInit_ex(), this function assumes that the context has
been pre-initialised and crashes when it isn't.
Bug: https://trac.libssh2.org/ticket/279Fixes#279
Commit d512b25f69a1b6778881f6b4b5ff9cfc6023be42 added automake
conditionals to Makefile.inc but since Makefile.inc is included
from Makefile for all other build systems that does not work.
This commit instead adds Makefile.OpenSSL.inc and Makefile.libgcrypt.inc
and moves the automake conditional to its proper place, src/Makefile.am.
The automake conditional includes the correct Makefile.$name.inc per
the crypto library selection/detection done by configure.
All non-autotools build system files in libssh2 are hardcoded to use
OpenSSL and do not get a conditional but at least there is some reuse
because they can all include the new Makefile.OpenSSL.inc.
The default channel window size used until now was 256KB. This value is
too small and results on a bottleneck on real-life networks where
round-trip delays can easily reach 300ms.
The issue was not visible because the configured channel window size
was being ignored and a hard-coded value of ~22MB being used instead,
but that was fixed on a previous commit.
This patch just changes the default window size
(LIBSSH2_CHANNEL_WINDOW_DEFAULT) to 2MB. It is the same value used by
OpenSSH and in our opinion represents a good compromise between memory
used and transfer speed.
Performance tests were run to determine the optimum value. The details
and related discussion are available from the following thread on the
libssh2 mailing-list:
http://www.libssh2.org/mail/libssh2-devel-archive-2013-10/0018.shtmlhttp://article.gmane.org/gmane.network.ssh.libssh2.devel/6543
An excerpt follows:
"I have been running some transfer test and measuring their speed.
My setup was composed of a quad-core Linux machine running Ubuntu 13.10
x86_64 with a LXC container inside. The data transfers were performed
from the container to the host (never crossing through a physical
network device).
Network delays were simulated using the tc tool. And ping was used to
verify that they worked as intended during the tests.
The operation performed was the equivalent to the following ssh command:
$ ssh container "dd bs=16K count=8K if=/dev/zero" >/dev/null
Though, establishment and closing of the SSH connection was excluded
from the timings.
I run the tests several times transferring files of sizes up to 128MB
and the results were consistent between runs.
The results corresponding to the 128MB transfer are available here:
https://docs.google.com/spreadsheet/ccc?key=0Ao1yRmX6PQQzdG5wSFlrZl9HRWNET3ZyN0hnaGo5ZFE&usp=sharing
It clearly shows that 256KB is too small as the default window size.
Moving to a 512MB generates a great improvement and after the 1MB mark
the returns rapidly diminish. Other factors (TCP window size, probably)
become more limiting than the channel window size
For comparison I also performed the same transfers using OpenSSH. Its
speed is usually on par with that of libssh2 using a window size of 1MB
(even if it uses a 2MB window, maybe it is less aggressive sending the
window adjust msgs)."
Signed-off-by: Salvador Fandino <sfandino@yahoo.com>
_libssh2_channel_read was using an arbitrary hard-coded limit to trigger
the window adjusting code. The adjustment used was also hard-coded and
arbitrary, 15MB actually, which would limit the usability of libssh2 on
systems with little RAM.
This patch, uses the window_size parameter passed to
libssh2_channel_open_ex (stored as remote.window_size_initial) plus the
buflen as the base for the trigger and the adjustment calculation.
The memory usage when using the default window size is reduced from 22MB
to 256KB per channel (actually, if compression is used, these numbers
should be incremented by ~50% to account for the errors between the
decompressed packet sizes and the predicted sizes).
My tests indicate that this change does not impact the performance of
transfers across localhost or a LAN, being it on par with that of
OpenSSH. On the other hand, it will probably slow down transfers on
networks with high bandwidth*delay when the default window size
(LIBSSH2_CHANNEL_WINDOW_DEFAULT=256KB) is used.
Signed-off-by: Salvador Fandino <sfandino@yahoo.com>
Store but don't use keys of unsupported types on the known_hosts file.
Currently, when libssh2 parses a known_host file containing keys of some
type it doesn't natively support, it stops reading the file and returns
an error.
That means, that the known_host file can not be safely shared with other
software supporting other key types (i.e. OpenSSH).
This patch adds support for handling keys of unknown type. It can read
and write them, even if they are never going to be matched.
At the source level the patch does the following things:
- add a new unknown key type LIBSSH2_KNOWNHOST_KEY_UNKNOWN
- add a new slot (key_type_name) on the known_host struct that is
used to store the key type in ascii form when it is not supported
- parse correctly known_hosts entries with unknown key types and
populate the key_type_name slot
- print correctly known_hosts entries of unknown type
- when checking a host key ignore keys that do not match the key
Fixes#276
Deflate may return Z_OK even when not all data has been compressed
if the output buffer becomes full.
In practice this is very unlikely to happen because the output buffer
size is always some KBs larger than the size of the data passed for
compression from the upper layers and I think that zlib never expands
the data so much, even on the worst cases.
Anyway, this patch plays on the safe side checking that the output
buffer is not exhausted.
Signed-off-by: Salvador <sfandino@yahoo.com>
The old algorithm was O(N^2), causing lots and lots of reallocations
when highly compressed data was transferred.
This patch implements a simpler one that just doubles the buffer size
everytime it is exhausted. It results in O(N) complexity.
Also a smaller inflate ratio is used to calculate the initial size (x4).
Signed-off-by: Salvador <sfandino@yahoo.com>
Data may remain in zlib internal buffers when inflate() returns Z_OK
and avail_out == 0. In that case, inflate has to be called again.
Also, once all the data has been inflated, it returns Z_BUF_ERROR to
signal that the input buffer has been exhausted.
Until now, the way to detect that a packet payload had been completely
decompressed was to check that no data remained on the input buffer
but that didn't account for the case where data remained on the internal
zlib buffers.
That resulted in packets not being completely decompressed and the
missing data reappearing on the next packet, though the bug was masked
by the buffer allocation algorithm most of the time and only manifested
when transferring highly compressible data.
This patch fixes the zlib usage.
Signed-off-by: Salvador <sfandino@yahoo.com>
After filling the read buffer with data from the read queue, when the
window size was too small, "libssh2_channel_receive_window_adjust" was
called to increase it. In non-blocking mode that function could return
EAGAIN and, in that case, the EAGAIN was propagated upwards and the data
already read on the buffer lost.
The function was also moving between the two read states
"libssh2_NB_state_idle" and "libssh2_NB_state_created" both of which
behave in the same way (excepting a debug statment).
This commit modifies "_libssh2_channel_read" so that the
"libssh2_channel_receive_window_adjust" call is performed first (when
required) and if everything goes well, then it reads the data from the
queued packets into the read buffer.
It also removes the useless "libssh2_NB_state_created" read state.
Some rotted comments have also been updated.
Signed-off-by: Salvador <sfandino@yahoo.com>
Until now, the window size (channel->remote.window_size) was being
updated just after receiving the packet from the transport layer.
That behaviour is wrong because the channel queue may grow uncontrolled
when data arrives from the network faster that the upper layer consumes
it.
This patch adds a new counter, read_avail, which keeps a count of the
bytes available from the packet queue for reading. Also, now the window
size is adjusted when the data is actually read by an upper layer.
That way, if the upper layer stops reading data, the window will
eventually fill and the remote host will stop sending data. When the
upper layers reads enough data, a window adjust packet is delivered and
the transfer resumes.
The read_avail counter is used to detect the situation when the remote
server tries to send data surpassing the window size. In that case, the
extra data is discarded.
Signed-off-by: Salvador <sfandino@yahoo.com>
libssh2 used to explicitly check for libgcrypt and default to OpenSSL.
Now all possible crypto libraries are checked for explicitly, making
the addition of further crypto libraries both simpler and cleaner.
Fixes issue arising when server does not support statfvs and or fstatvfs
extensions. sftp_statvfs() and sftp_fstatvfs() after this patch will
handle the case when SSH_FXP_STATUS is returned from server.
This partially reverts commit 03ca9020756a4e16f0294e5b35e9826ee6af2364
in order to fix extreme slowdown when uploading to localhost via SFTP.
I was able to repeat the issue on RHEL-7 on localhost only. It did not
occur when uploading via network and it did not occur on a RHEL-6 box
with the same version of libssh2.
The problem was that sftp_read() used a read-ahead logic to figure out
the window_size, but sftp_packet_read() called indirectly from
sftp_write() did not use any read-ahead logic.
When there's no window to "write to", there's no point in waiting for
the socket to become writable since it most likely just will continue to
be.
Patch-by: ncm
Fixes#258
In _libssh2_packet_add, called by _libssh2_packet_read, a call to
_libssh2_packet_send that is supposed to send a one-byte message
SSH_MSG_REQUEST_FAILURE would send an uninitialized byte upon re-entry
if its call to _send returns _EAGAIN.
Fixes#259
The new libssh2_sftp_fsync API causes data and metadata in the
currently open file to be committed to disk at the server.
This is an OpenSSH extension to the SFTP protocol. See:
https://bugzilla.mindrot.org/show_bug.cgi?id=1798
... in macro parameters to avoid compiler warnings about lost precision.
Several macros in libssh2.h call strlen and pass the result directly to
unsigned int parameters of other functions, which warns about precision
loss because strlen returns size_t which is unsigned long on at least
some platforms (such as OS X). The fix is to simply typecast the
strlen() result to unsigned int.
libssh2_knownhost_readfile() silently ignored problems when reading keys
in unsupported formats from the known hosts file. When the file is
written again from the internal structures of libssh2 it gets truntcated
to the point where the first unknown key was located.
* src/knownhost.c:libssh2_knownhost_readfile() - return error if key
parsing fails
Add a "use_in_auth" flag to the LIBSSH2_COMP_METHOD struct and a
separate "zlib@openssh.com" method, along with checking session->state
for LIBSSH2_STATE_AUTHENTICATED. Appears to work on the OpenSSH servers
I've tried against, and it should work as before with normal zlib
compression.
If libssh2_session_free is called without the channel being freed
previously by libssh2_channel_free a memory leak could occur.
A mismatch of states variables in session_free() prevent the call to
libssh2_channel_free function. session->state member is used instead of
session->free_state.
It causes a leak of around 600 bytes on every connection on my systems
(Linux, x64 and PPC).
(Debugging done under contract for Accedian Networks)
Fixes#246
When using libssh2 to perform an SFTP file transfer from the "JSCAPE MFT
Server" (http://www.jscape.com) the transfer failed. The default JSCAPE
configuration is to enforce zlib compression on SSH2 sessions so the
session was compressed. The relevant part of the debug trace contained:
[libssh2] 1.052750 Transport: unhandled zlib error -5
[libssh2] 1.052750 Failure Event: -29 - decompression failure
The trace comes from comp_method_zlib_decomp() in comp.c. The "unhandled
zlib error -5" is the status returned from the zlib function
inflate(). The -5 status corresponds to "Z_BUF_ERROR".
The inflate() function takes a pointer to a z_stream structure and
"inflates" (decompresses) as much as it can. The relevant fields of the
z_stream structure are:
next_in - pointer to the input buffer containing compressed data
avail_in - the number of bytes available at next_in
next_out - pointer to the output buffer to be filled with uncompressed
data
avail_out - how much space available at next_out
To decompress data you set up a z_stream struct with the relevant fields
filled in and pass it to inflate(). On return the fields will have been
updated so next_in and avail_in show how much compressed data is yet to
be processed and next_out and avail_out show how much space is left in
the output buffer.
If the supplied output buffer is too small then on return there will be
compressed data yet to be processed (avail_in != 0) and inflate() will
return Z_OK. In this case the output buffer must be grown, avail_out
updated and inflate() called again.
If the supplied output buffer was big enough then on return the
compressed data will have been exhausted (avail_in == 0) and inflate()
will return Z_OK, so the data has all been uncompressed.
There is a corner case where inflate() makes no progress. That is, there
may be unprocessed compressed data and space available in the output
buffer and yet the function does nothing. In this case inflate() will
return Z_BUF_ERROR. From the zlib documentation and the source code it
is not clear under what circumstances this happens. It could be that it
needs to write multiple bytes (all in one go) from its internal state to
the output buffer before processing the next chunk of input but but
can't because there is not enough space (though my guesses as to the
cause are not really relevant). Recovery from Z_BUF_ERROR is pretty
simple - just grow the output buffer, update avail_out and call
inflate() again.
The comp_method_zlib_decomp() function does not handle the case when
inflate() returns Z_BUF_ERROR. It treats it as a non-recoverable error
and basically aborts the session.
Fixes#240
Changed to use Windows commandline tools instead of
GNU tools when compiling on Windows. Fixed dist and
dev targets. Enabled nlmconv error for unresolved
symbols.
Usually a format macro should hold the whole format, otherwise
it should be named a prefix. Also fixed usage of this macro in
scp.c for a signed var where it was used as prefix for unsigned.
Rationale: Everything else in this file states a fact about the win32
platform that is unconditional for that platform. There is nothing
unconditional about the presence of zlib. It is neither included with
Windows nor with the platform SDK. Therefore, this is not an appropriate
place to assert its presence. Especially as, once asserted, it cannot be
overridden using a compiler flag.
In contrast, if it is omitted, then it can easily be reasserted by adding
a compiler flag defining LIBSSH2_HAVE_ZLIB.
sftp_packet_add takes ownership of the packet passed to it and (now that we
handle zombies) might free the packet. sftp_packet_read uses the packet type
byte as its return code but by this point sftp_packet_add might have freed
it. This change fixes the problem by caching the packet type before calling
sftp_packet_add.
I don't understand why sftp_packet_read uses the packet type as its return
code. A future change might get rid of this entirely.
As this function is called when the SFTP session is closed, it needs to
also kill all zombies left in the SFTP session to avoid leaking memory
just in case some zombie would still be in there.
When flushing the packetlist, we must only add the request as a zombie
if no response has already been received. Otherwise we could wrongly
make it a zombie even though the response was already received and then
we'd get a zombie stuck there "forever"...
Since the sftp_packetlist_flush() function will move all the existing
FXP_READ requests in this handle to the zombie list we must first remove
this just received packet as it is clearly not a zombie.
Exactly as the comment in the code said, checking the return code from
sftp_packet_read() with <= was wrong and it should be < 0. With the new
filtering on incoming packets that are "zombies" we can now see this
getting zero returned.
In order to be fast, sftp_read sends many read requests at once. With a small
file, this can mean that when EOF is received back, many of these requests are
still outstanding. Responses arriving after we close the file and abandon the
file handle are queued in the SFTP packet queue and never collected. This
causes transfer speed to drop as a progressively longer queue must be searched
for every packet.
This change introduces a zombie request-ID list in the SFTP session that is
used to recognise these outstanding requests and prevent them being added to
the queue.
libcrypto on win32 now depends on gdi32.dll, so move the OpenSSL LDLIBS
block to before the compiler definitions, so that libcrypto gets added
first, and then add -lgdi32 into the following common LDLIBS for gcc.
Examples are built by default. Any of the following options on the
configure command line will skip building them:
--disable-examples-build
--enable-examples-build=no
--enable-examples-build=false
If the filename parameter for file_read_publickey() was the name of a
directory instead of a file then libssh2 would spin trying to fgetc()
from the FILE * for the opened directory when trying to determine the
length of the encoded public key, since fgetc() can't report errors.
Use fread() instead to correctly detect this error condition along
with many others.
This fixes the problem reported in
http://www.libssh2.org/mail/libssh2-devel-archive-2012-04/0021.shtml
Reported-by: Oleksiy Zagorskyi <zalex_ua@i.ua>
When calling _libssh2_channel_receive_window_adjust() internally, we now
always use the 'force' option to prevent libssh2 to avoid sending the
update if the update isn't big enough.
It isn't fully analyzed but we have seen corner cases which made a
necessary window update not get send due to this and then the other side
doesn't send data our side then sits waiting for forever.
if there's not enough room to receive the data that's being requested,
the window adjustment needs to be sent to the remote and thus the force
option has to be used. _libssh2_channel_receive_window_adjust() would
otherwise "queue" small window adjustments for a later packet but that
is really terribly for the small buffer read that for example is the
final little piece of a very large file as then there is no logical next
packet!
Reported by: Armen Babakhanian
Bug: http://www.libssh2.org/mail/libssh2-devel-archive-2012-03/0130.shtml
_libssh2_channel_write() first reads outstanding packets before writing
new data. If it reads a key exchange request, it will immediately start
key re-exchange, which will require sending a response. If the output
socket is full, this will result in a return from
_libssh2_transport_read() of LIBSSH2_ERROR_EAGAIN. In order not to block
a write because there is no data to read, this error is explicitly
ignored and the code continues marshalling a packet for sending. When it
is sent, the remote end immediately drops the connection because it was
expecting a continuation of the key exchange, but got a data packet.
This change adds the same check for key exchange to
_libssh2_transport_send() that is in _libssh2_transport_read(). This
ensures that key exchange is completed before any data packet is sent.
When draining data off the socket with _libssh2_transport_read() (which
in turn has to be done so that we can be sure to have read any possible
window-increasing packets), this code previously ignored errors which
could lead to nasty loops. Now all error codes except EAGAIN will cause
the error to be returned at once.
Bug: http://www.libssh2.org/mail/libssh2-devel-archive-2012-03/0068.shtml
Reported by: Matthew Booth
sizeof(buf) expands to 8 or 4 (since its a pointer). This variable may
have been static in the past, leading to this error.
Signed-off-by: Steven Dake <sdake@redhat.com>
In the x11 example, sizeof(buf) = 8UL (on x86_64), when this should
probably represent the buffer size available. I am not sure how to
test that this change is actually correct, however.
Signed-off-by: Steven Dake <sdake@redhat.com>
The commit in 7194a9bd7ba45 wasn't complete. This change makes sure
variables are initialized properly before used in the EAGAIN and window
adjust cases.
Lots of places in the code translated the original error into the more
generic LIBSSH2_ERROR_SOCKET_TIMEOUT but this turns out to distort the
original error reason a lot and makes tracking down the real origin of a
problem really hard. This change makes the original error code be
preserved to a larger extent when return up to the parent function.
Commit 03ca9020756 tried to simplify the window sizing logic but broke
SFTP readdir as there was no window sizing code left there so large
directory listings no longer worked.
This change introduces window sizing logic to the sftp_packet_read()
function so that it now tells the remote about the local size having a
window size that suffice when it is about to ask for directory data.
Bug: http://www.libssh2.org/mail/libssh2-devel-archive-2012-03/0069.shtml
Reported by: Eric
The call of libssh2_init returns a return code, but nothing could be done
within the _libssh2_init_if_needed execution path.
Signed-off-by: Steven Dake <sdake@redhat.com>
Although the function checks the length, if the code was in error, there
could potentially be a buffer overrun with the use of sprintf. Instead replace
with snprintf.
Signed-off-by: Steven Dake <sdake@redhat.com>
INVALID_SOCKET is a special value in Windows representing a
non-valid socket identifier. We were #defining this to -1 on
non-Windows platforms, causing unneccessary namespace pollution.
Let's have our own identifier instead.
Thanks to Matt Lawson for pointing this out.
OpenSolaris has no cfmakeraw() so to make the example more portable
we simply do the equivalent operations on struct termios ourselves.
Thanks to Tom Weber for reporting this problem, and finding a solution.
Whenever we have acked data and is about to call a function that *MAY*
return EAGAIN we must return the number now and wait to get called
again. Our API only allows data *or* EAGAIN and we must never try to get
both.
Removed the total_read variable that originally must have tracked how
much data had been written to the buffer. With non-blocking reads, we
must return straight away once we have read data into the buffer so this
variable served not purpose.
I think it was still hanging around in case the initial processing of
'leftover' data meant we wrote to the buffer but this case, like the
others, must return immediately. Now that it does, the last remaining
need for the variable is gone.
Whenever we have data and is about to call a function that *MAY* return
EAGAIN we must return the data now and wait to get called again. Our API
only allows data *or* EAGAIN and we must never try to get both.
Commit 209de22299b4b58e582891dfba70f57e1e0492db introduced a function
call to a non-existing function, and since then the libgcrypt backend
has not been buildable.
If the function that extracts/computes the public key from a private key
fails the errors it reports were masked by the function calling it. This
patch modifies the key extraction function to return errors using
_libssh_error() function. The error messages are tweaked to contain
reference to the failed operaton in addition to the reason.
* AUTHORS: - add my name
* libgcrypt.c: _libssh2_pub_priv_keyfile(): - return a more verbose
error using
_libssh2_error() func.
* openssl.c: - modify call graph of _libssh2_pub_priv_keyfile() to use
_libssh2_error for error reporting();
* userauth.c: - tweak functions calling _libssh2_pub_priv_keyfile() not
to shadow error messages
Some SFTP servers send SFTP packets larger than 40000. Since the limit
is only present to avoid insane sizes anyway, we can easily bump it.
The define was formerly in the public header libssh2_sftp.h but served
no external purpose and was moved into the source dir.
Bug: http://www.libssh2.org/mail/libssh2-devel-archive-2011-11/0004.shtml
Reported by: Michael Harris
Documentation for libssh2_knownhost_checkp() and related functions
states that the last argument is filled with data if non-NULL.
"knownhost if set to non-NULL, it must be a pointer to a 'struct
libssh2_knownhost' pointer that gets filled in to point to info about a
known host that matches or partially matches."
In this function ext is dereferenced even if set to NULL, causing
segfault in applications not needing the extra data.
In function knownhost_add, memory is alocated for a new entry. If normal
alocation is used, memory is not initialized to 0 right after, but a
check is done to verify if correct key type is passed. This test is done
BEFORE setting the memory to null, and on the error path function
free_host() is called, that tries to dereference unititialized memory,
resulting into a glibc abort().
* knownhost.c - knownhost_add(): - move typemask check before alloc
When cross compiling to Windows, libssh2.h include Windows header files
with upper case filenames : BaseTsd.h and WinSock2.h.
These files have lowercase names with mingw-w64 (iirc, it's the same with
mingw). And as on Windows, being lowercase or uppercase does not matter.
Make sure we don't clear or reset static structs after first init so
that they work fine even when used from multiple threads. Init the
structs in the global init.
Help and assistance by: John Engstrom
Fixes#229 (again)
make_ctr_evp() is changed to take a struct pointer, and then each
_libssh2_EVP_aes_[keylen]_ctr function is made to pass in their own
static struct
Reported by: John Engstrom
Fixes#229
Set read_state back to idle before trying to send anything so that if
the state somehow is wrongly set.
Also, avoid such a case of confusion by resetting the read_state when an
sftp handle is closed.
Removed the automatic window_size adjustments from
_libssh2_channel_read() and instead all channel readers must now make
sure to enlarge the window sizes properly themselves.
libssh2_channel_read_ex() - the public function, now grows the window
size according to the requested buffer size. Applications can still opt
to grow the window more on demand. Larger windows tend to give higher
performance.
sftp_read() now uses the read-ahead logic to figure out a window_size.
A returned READ packet that is short will now only reduce the
offset.
This is a temporary fix as it is slightly better than the previous
approach but still not very good.
When receiving more data than what the window size allows on a
particular channel, make sure that the window size is adjusted in that
case too. Previously it would only adjust the window in the non-error
case.
When seeking to a new position, flush the packetlist and buffered data
to prevent already received or pending data to wrongly get used when
sftp-reading from the new offset within the file.
In the case where a read packet has been received from the server, but
the entire contents couldn't be copied to the user-buffer, the data is
instead buffered and copied to the user's buffer in the next invocation
of sftp_read(). When that "extra" copy is made, the 'offset' pointer was
not advanced accordingly.
The biggest impact of this flaw was that the 'already' variable at the
top of the function that figures out how much data "ahead" that has
already been asked for would slowly go more and more out of sync, which
could lead to the file not being read all the way to the end.
This problem was most noticable in cases where the application would
only try to read the exact file size amount, like curl does. In the
examples libssh2 provides the sftp read function is most often called
with a fixed size large buffer and then the bug would not appear as
easily.
This bug was introduced in the SFTP rewrite in 1.2.8.
Bug: http://curl.haxx.se/mail/lib-2011-08/0305.htmlhttp://www.libssh2.org/mail/libssh2-devel-archive-2011-08/0085.shtml
Partly reverse 566894494b4972ae12 which was simplifying the code far too
much and ended up overflowing a buffer within the LIBSSH2_SESSION
struct. Back to allocating the buffer properly like it used to do.
Bug: http://www.libssh2.org/mail/libssh2-devel-archive-2011-06/0032.shtml
Reported by: Alfred Gebert
A sftp session failed with error "failure establishing ssh session" on
Solaris and HP-UX. Sometimes the first recv() function call sets errno
to ENOENT. In the man pages for recv of Solaris and HP-UX the error
ENOENT is not documented.
I tested Solaris SPARC and x86, HP-UX i64, AIX, Windows and Linux.
After an error occurs in libssh2_scp_recv() or libssh2_scp_send(), the
function libssh2_session_last_error() would return
LIBSSH2_ERROR_SOCKET_NONE on error.
Bug: http://trac.libssh2.org/ticket/216
Patch by: "littlesavage"
Fixes#216
Added libraries needed to link whether using openssl dynamically or
statically
Added LIBSSH2DEBUG define to debug versions to enable tracing
URL: http://trac.libssh2.org/ticket/215
Patch by: Mark Smith
Someone on IRC pointed out that we don't have these documented so I
wrote up a first set based on the information in the wiki:
http://trac.libssh2.org/wiki/KeepAlive
Stop using the $VERSION variable as it seems to be magically used by
autoconfig itself and thus gets set to the value set in AC_INIT()
without us wanting that. $LIBSSH2VER is now the libssh2 version as
detected.
Reported by: Paul Howarth
Bug: http://www.libssh2.org/mail/libssh2-devel-archive-2011-04/0008.shtml
Starting now, the NEWS file is generated from git using the git2news.pl
script. This makes it always accurate and up-to-date, even for daily
snapshots etc.
Previously the code assumed either a single host name or a hostname,ip-address pair. However, according to the spec [1], there can be any number of comma separated host names or IP addresses.
[1] http://www.openbsd.org/cgi-bin/man.cgi?query=sshd&sektion=8
Fix the bug that libssh2 could not connect if the sftp server
sends data before sending the version string.
http://tools.ietf.org/html/rfc4253#section-4.2
"The server MAY send other lines of data before sending the version
string. Each line SHOULD be terminated by a Carriage Return and Line
Feed. Such lines MUST NOT begin with "SSH-", and SHOULD be encoded
in ISO-10646 UTF-8 [RFC3629] (language is not specified). Clients
MUST be able to process such lines."
The buffer for the decompression (remote.comp_abstract) is initialised
in time when it is needed. With this fix decompression is disabled when
the buffer (remote.comp_abstract) is not initialised.
Bug: http://trac.libssh2.org/ticket/200
As pointed out in bug #206, if a second invoke of libssh2_sftp_read()
would shrink the buffer size, libssh2 would go nuts and send out read
requests like crazy. This was due to an unsigned variable turning
"negative" by some wrong math, and that value would be the amount of
data attempt to pre-buffer!
Bug: http://trac.libssh2.org/ticket/206
If asked to read data into a buffer and the buffer is too small to hold
the data, this function now returns an error instead of as previously
just copy as much as fits.
As discussed on the mailing list, it was wrong for win64 and using the
VC-provided type is the safest approach instead of second- guessing
which one it should be.
Added crypto.h that is the unified header to include when using crypto
functionality. It should be the only header that needs to adapt to the
underlying crypto library in use. It provides the set of prototypes that
are library agnostic.
Pass a NULL pointer for the publickey parameter of
libssh2_userauth_publickey_fromfile and
libssh2_userauth_hostbased_fromfile functions. In this case, the
functions recompute the public key from the private key file data.
This is work done by Jean-Louis CHARTON
<Jean-Louis.CHARTON@oikialog.com>, then adapted by Mark Smith and
slightly edited further by me Daniel.
WARNING: this does leave the feature NOT WORKING when libssh2 is built
to use libgcrypt instead of OpenSSL simply due to lack of
implementation.
By using a new separate struct for incoming SFTP packets and not sharing
the generic packet struct, we can get rid of an unused field and add a
new one dedicated for holding the request_id for the incoming
package. As sftp_packet_ask() is called fairly often, a "mere" integer
comparison is MUCH faster than the previous memcmp() of (typically) 5
bytes.
Make sure that we cleanup remainders when the handle is closed and when
the subsystem is shutdown.
Existing flaw: if a single handle sends packets that haven't been
replied to yet at the time when the handle is closed, those packets will
arrive later and end up in the generic packet brigade queue and they
will remain in there until flushed. They will use unnecessary memory,
make things slower and they will ruin the SFTP handling if the
request_id counter ever wraps (highly unlikely to every happen).
The SFTP read function now does transfers the same way the SFTP write
function was made to recently: it creates a list of many outgoing
FXP_READ packets that each asks for a small data chunk. The code then
tries to keep sending read request while collecting the acks for the
previous requests and returns the received data.
The loop that waits for remote.close to get set may end up looping
forever since session->socket_state gets set to
LIBSSH2_SOCKET_DISCONNECTED by the packet_add() function called from the
transport_read() function and after having been set to
LIBSSH2_SOCKET_DISCONNECTED, the transport_read() function will only
return 0.
Bug: http://trac.libssh2.org/ticket/198
Split off libssh2_sftp_seek64 from the libssh2_sftp_seek man page, and
mentioned that we consider the latter deprecated. Also added a mention
about the dangers of doing seek during writing or reading.
The new SFTP write code caused a regression as the seek function no
longer worked as it didn't set the write position properly.
It should be noted that seeking is STRONGLY PROHIBITED during upload, as
the upload magic uses two different offset positions and the multiple
outstanding packets etc make them sensitive to change in the midst of
operations.
This functionality was just verified with the new example code
sftp_append. This bug was filed as bug #202:
Bug: http://trac.libssh2.org/ticket/202
I ran SFTP upload tests against localhost. It showed that to make the
app reach really good speeds, I needed to do a little code tweak and
change MAX_SFTP_OUTGOING_SIZE from 4000 to 30000. The tests I did before
with the high latency tests didn't show any real difference whatever I
had that size set to.
This number is the size in bytes that libssh2 cuts off the large input
buffer and sends off as an individual sftp packet.
This is an example that is very similar to sftp_write_nonblock.c, with
the exception that this uses
1 - a larger upload buffer
2 - a sliding buffer mechnism to allow the app to keep sending lots of
data to libssh2 without having to first drain the buffer.
These are two key issues to make libssh2 SFTP uploads really perform
well at this point in time.
The attempts made to have _libssh2_channel_write() accept larger pieces
of data and split up the data by itself into 32700 byte chunks and pass
them on to channel_write() in a loop as a way to do faster operations on
larger data blocks was a failed attempt.
The reason why it is difficult:
The API only allows EAGAIN or a length to be returned. When looping over
multiple blocks to get sent, one block can get sent and the next might
not. And yet: when transport_send() has returned EAGAIN we must not call
it again with new data until it has returned OK on the existing data it
is still working on. This makes it a mess and we do get a much easier
job by simply returning the bytes or EAGAIN at once, as in the EAGAIN
case we can assume that we will be called with the same arguments again
and transport_send() will be happy.
Unfortunately, I think we take a small performance hit by not being able
to do this.
This is a new example snippet. The code is largely based on ssh2_exec,
and is written by Tommy Lindgren. I edited it into C90 compliance and to
conform to libssh2 indent style and some more.
When a piece of data is sent from the send_existing() function we must
make the parent function return afterwards. Otherwise we risk that the
parent function tries to send more data and ends up getting an EGAIN for
that more data and since it can only return one return code it doesn't
return info for the successfully sent data.
As this change is a regression I now added a larger comment explaining
why it has to work like this.
Starting now, we unconditionally use the internal replacement functions
for send() and recv() - creatively named _libssh2_recv() and
_libssh2_send().
On errors, these functions return the negative 'errno' value instead of
the traditional -1. This design allows systems that have no "natural"
errno support to not have to invent it. It also means that no code
outside of these two transfer functions should use the errno variable.
Some checks are better done in _libssh2_channel_write just once per
write instead of in channel_write() since the looping will call the
latter function multiple times per _libssh2_channel_write() invoke.
The SFTP handle struct now buffers number of acked bytes that haven't
yet been returned. The way this is used is as following:
1. sftp_write() gets called with a buffer of let say size 32000. We
split 32000 into 8 smaller packets and send them off one by one. One of
them gets acked before the function returns so 4000 is returned.
2. sftp_write() gets called again a short while after the previous one,
now with a much smaller size passed in to the function. Lets say 8000.
In the mean-time, all of the remaining packets from the previous call
have been acked (7*4000 = 28000). This function then returns 8000 as all
data passed in are already sent and it can't return any more than what
it got passed in. But we have 28000 bytes acked. We now store the
remaining 20000 in the handle->u.file.acked struct field to add up in
the next call.
3. sftp_write() gets called again, and now there's a backlogged 20000
bytes to return as fine and that will get skipped from the beginning
of the buffer that is passed in.
When SCP send or recv fails, it gets a special message from the server
with a warning or error message included. We have no current API to
expose that message but the foundation is there. Removed unnecessary use
of session struct fields.
I added size checks in several places. I fixed the code flow to be easier
to read in some places.
I removed unnecessary zeroing of structs. I removed unused struct fields.
We don't like magic numbers in the code. Now the acceptable failure
codes sent in the SSH_MSG_CHANNEL_OPEN_FAILURE message are added as
defined values in the private header file.
This function now only returns EAGAIN if a lower layer actually returned
EAGAIN to it. If nothing was acked and no EAGAIN was received, it will
now instead return 0.
If _libssh2_wait_socket() gets called but there's no direction set to
wait for, this causes a "hang". This code now detects this situation,
set a 1 second timeout instead and outputs a debug output about it.
SFTP packets come as [32 bit length][payload] and the code didn't
previously handle that the initial 32 bit field was read only partially
when it was read.
While setting up the session, ssh tries to determine the type of
encryption method it can use for the session. This requires looking at
the keys offered by the remote host and comparing these with the methods
supported by libssh2 (rsa & dss). To do this there is an iteration over
the array containing the methods supported by libssh2.
If there is no agreement on the type of encryption we come to the 3rd
entry of the hostkeyp array. Here hostkeyp is valid but *hostkep is
NULL. Thus when we dereference that in (*hostkeyp)->name there is a
crash
There were some chances that they would cause -1 to get returned by
public functions and as we're hunting down all such occurances and since
the underlying functions do return valuable information the code now
passes back proper return codes better.
The man page clearly says it returns 1 for "already authenticated" but
the code said non-zero. I changed the code to use 1 now, as that is also
non-zero but it gets the benefit that it now matches the documentation.
Using 1 instead of non-zero is better for two reasons:
1. We have the opportunity to introduce other return codes in the future for
things like error and what not.
2. We don't expose the internal bitmask variable value.
First I wanted to free the memory in session_free() but then
I had still memory leaks because in my test case the function
userauth_keyboard_interactive() is called twice. It is called
twice perhaps because the server has this authentication
methods available: publickey,gssapi-with-mic,keyboard-interactive
The keyboard-interactive method is successful.
I found an undocumented public function and we can't have it like
that. The description here is incomplete, but should serve as a template
to allow filling in...
The sftp_write function shouldn't assume that the buffer pointer will be
the same in subsequent calls, even if it assumes that the data already
passed in before haven't changed.
The sftp structs are now moved to sftp.h (which I forgot to add before)
sftp_write was rewritten to split up outgoing data into multiple packets
and deal with the acks in a more asynchronous manner. This is meant to
help overcome latency and round-trip problems with the SFTP protocol.
Neither _libssh2_channel_write nor sftp_write now have the 32500 size
limit anymore and instead the channel writing function now has its own
logic to send data in multiple calls until everything is sent.
The new function takes two data areas, combines them and sends them as a
single SSH packet. This allows several functions to allocate and copy
less data.
I also found and fixed a mixed up use of the compression function
arguments that I introduced in my rewrite in a recent commit.
We now allow libssh2_session_flag() to enable compression with a new
flag and I added documentation for the previous LIBSSH2_FLAG_SIGPIPE
flag which I wasn't really aware of!
As a zero return code from channel_read() is not an error we must make
sure that the SCP functions deal with that properly. channel_read()
always returns 0 if the channel is EOFed already so we check for EOF
after 0-reads to be able to return error properly.
In the transport functions we avoid a strcmp() now and just check a
boolean instead.
The compress/decompress function's return code is now acknowledged and
used as actual return code in case of failures.
The function libssh2_session_startup() is now considered deprecated due
to the portability issue with the socket argument.
libssh2_session_handshake() is the name of the replacement.
When a call to _libssh2_transport_write() succeeds, we must return from
_libssh2_channel_write() to allow the caller to provide the next chunk
of data.
We cannot move on to send the next piece of data that may already have
been provided in this same function call, as we risk getting EAGAIN for
that and we can't return information both about sent data as well as
EAGAIN. So, by returning short now, the caller will call this function
again with new data to send.
Replaced -1/SOCKET_NONE errors with appropriate error defines instead.
Made the verbose trace output during banner receiving less annoying for
non-blocking sessions.
In an attempt to make the trace output less cluttered for non-blocking
sessions the error function now avoids calling the debug function if the
error is the EAGAIN and the session is non-blocking.
LIBSSH2_SOCKET_NONE (-1) should no longer be used as error code as it is
(too) generic and we should instead use specific and dedicated error
codes to better describe the error.
The well known and used ssh server Dropbear has a maximum SSH packet
length at 32768 by default. Since the libssh2 design current have a
fixed one-to-one mapping from channel_write() to the packet size created
by transport_write() the previous limit of 32768 in the channel layer
caused the transport layer to create larger packets than 32768 at times
which Dropbear rejected forcibly (by closing the connection).
The long term fix is of course to remove the hard relation between the
outgoing SSH packet size and what the input length argument is in the
transport_write() function call.
When parsing the SCP protocol and verifying that the data looks like a
valid file name, byte values over 126 must not be consider illegal since
UTF-8 file names will use such codes.
Reported by: Uli Zappe
Bug: http://www.libssh2.org/mail/libssh2-devel-archive-2010-08/0112.shtml
No idea why we had this ifdef at all but MSVC, MingW32, Watcom
and Borland all have no sys/uio.h header; so if there's another
Win32 compiler which needs it then it should be added explicitely
instead of this negative list.
Since libssh2 often sets LIBSSH2_ERROR_EAGAIN internally before
_libssh2_wait_socket is called, we can decrease some amount of
confusion in user programs by resetting the error code in this function
to reduce the risk of EAGAIN being stored as error when a blocking
function returns.
As reported on the mailing list, the code path using poll() should
multiple seconds with 1000 to get milliseconds, not divide!
Reported by: Jan Van Boghout
The condition around the ssize_t typedef depended on both LIBSSH2_WIN32
*and* _MSC_VER being defined when it should be enough to depend on
_MSC_VER only. It also makes it nicer so libssh2-using code builds fine
without having custom defines.
As was pointed out in bug #182, we must not return failure from
_libssh2_channel_free() when _libssh2_channel_close() returns an error
that isn't EAGAIN. It can effectively cause the function to never go
through, like it did now in the case where the socket was actually
closed but socket_state still said LIBSSH2_SOCKET_CONNECTED.
I consider this fix the right thing as it now also survives other
errors, even if making sure socket_state isn't lying is also a good
idea.
Fixed a compiler warning I introduced previously when checking input
arguments more. I also added a check for the other pointer to avoid NULL
pointer dereferences.
Make use of the EVP interface for the AES-funktion. Using this method
supports the use of different ENGINES in OpenSSL for the AES function
(and the direct call to the AES_encrypt should not be used according to
openssl.org)
There was a buffer overflow waiting to happen when a debug message was
longer than 1536 bytes.
Thanks to Daniel who spotted that there was a problem with the message
length passed to a trace handler also after commit
0f0652a3093111fc7dac0205fdcf8d02bf16e89f.
In KEXINIT messages, the client and server agree on, among other
things, whether to use compression. This method agreement occurs
in src/kex.c's kex_agree_methods() function. However, if
compression is enabled (either client->server, server->client, or
both), then the compression layer is initialized in
kex_agree_methods() -- before NEWKEYS has been received.
Instead, the initialization of the compression layer should
happen after NEWKEYS has been received. This looks to occur
insrc/kex.c's diffie_hellman_sha1(), which even has the comment:
/* The first key exchange has been performed,
switch to active crypt/comp/mac mode */
There, after NEWKEYS is received, the cipher and mac algorithms
are initialized, and that is where the compression should be
initialized as well.
The current implementation fails if server->client compression is
enabled because most server implementations follow OpenSSH's
lead, where compression is initialized after NEWKEYS. Since the
server initializes compression after NEWKEYS, but libssh2
initializes compression after KEXINIT (i.e. before NEWKEYS), they
are out of sync.
Reported in bug report #180
The packet length calculated in src/userauth.c's
userauth_hostbased_fromfile() function is too short by 4 bytes;
it forgets to add four bytes for the length of the hostname.
This causes hostbased authentication to fail, since the server
will read junk data.
verified against proftpd's mod_sftp module
This functions get the method length by looking at the first 32
bit of data, and I now made it not accept method lengths that are
longer than the whole data set is, as given in the dedicated
function argument.
This was detected when the function was given bogus public key
data as an ascii string, which caused the first 32bits to create
a HUGE number.
Sending in NULL as the primary pointer is now dealt with by more
public functions. I also narrowed the userauth.c code somewhat to
stay within 80 columns better.
The LIBSSH2_DEBUG macro, defined in libssh2_priv.h, incorrectly uses the
function variable ssh_msg_disconnect when it should use ssh_msg_debug.
This shows that the LIBSSH2_CALLBACK_DEBUG callback never has worked...
The individual man pages for macros now show the full convenience
macro as defined, and then the man page for the actual function
only shows the function.
sftp_write() now limits how much data it gets at a time even more
than before. Since this function creates a complete outgoing
packet based on what gets passed to it, it is crucial that it
doesn't create too large packets.
With this method, there's also no longer any problem to use very
large buffers in your application and feed that to libssh2. I've
done numerous tests now with uploading data over SFTP using 100K
buffers and I've had no problems with that.
The select() is just to make it nicer so that it doesn't
crazy-loop on EAGAIN. The buffer size thing is mostly to verify
that this really work as supposed.
Transfer timing is just a minor thing, but it can just as well be
there and help us time and work on performance easier using out
of the box examples.
As pointed out in bug report #173, this module basically never
used _libssh2_error() which made it work inconstently with other
parts of the libssh2 code base. This is my first take at making
this code more in line with the rest.
If an application accidentally provides a NULL handle pointer to
the channel or sftp public functions, they now return an error
instead of segfaulting.
'last_errno' holds to the error code from the SFTP protocol and
since that is 32 bits on the wire there's no point in using a
long for this internally which is larger on some platforms.
agent->ops gets initialized by the libssh2_agent_connect() call
but we need to make sure that we don't segfault even if a bad
sequence of function calls is used.
Passing an invalid public key to libssh2_userauth_publickey_fromfile_ex
triggered an assertion. Replaced this with a runtime check that rejects
obviously invalid key data.
Alexander Lamaison filed bug #172
(http://trac.libssh2.org/ticket/172), and pointed out that SFTP
init would do bad if the session isn't yet authenticated at the
time of the call, so we now check for this situation and returns
an error if detected. Calling sftp_init() at this point is bad
usage to start with.
In order to increase portability of this example, I'm bringing
the inclusion of libssh2_config.h back, and I also added an
require that header for this example to compile.
I also made all code lines fit within 80 columns.
As the long-term goal is to get rid of the extensive set of
macros from the API we can just as well start small by not adding
new macros when we add new functions. Therefore we let the
function be libssh2_sftp_statvfs() plainly without using an _ex
suffix.
I also made it use size_t instead of unsigned int for the string
length as that too is a long-term goal for the API.
As pointed out by Grubsky Grigory <g.grubsky@securitycode.ru>, I
made a mistake when I added the _libssh2_store_str() call before
and I made a slightly different patch than what he suggested.
Based purely on taste.
libssh2_knownhost_checkp took 0 as a magic port number that indicated
a 'generic' check should be performed. However, 0 is a valid port
number in its own right so this commit changes the magic value to any
negative int.
OpenSSH has ways to add hosts to the knownhosts file that include
a specific port number which makes the key associated with only
that specific host+port pair. libssh2 previously did not support
this, and I was forced to add a new function to the API to
properly expose this ability to applications:
libssh2_knownhost_checkp()
To *add* such hosts to the knownhosts file, you make sure to pass
on the host name in that manner to the libssh2_knownhost_addc()
function.
There was some stub-like parts of an implementation for
implementing kex language negotiation that caused clang-analyzer
to warn and as it did nothing I've now removed the dead code.
The clang-analyzer report made it look into this function and
I've went through it to remove a potential use of an
uninitialized variable and I also added some validation of input
data received from the server.
In general, lots of more code in this file need to validate the
input before assuming it is correct: there are servers out there
that have bugs or just have another idea of how to do the SFTP
protocol.
The errors mentioned in this man page are possible return codes
but not necessarily the only return codes that this can return.
Also reformatted the typ prototypes somewhat.
To get the blocking vs non-blocking to work as smooth as possible
and behave better internally, we avoid using the external
interfaces when calling functions internally.
Renamed a few internal functions to use _libssh2 prefix when not
being private within a file, and removed the libssh2_ for one
that was private within the file.
This was triggered by a clang-analyzer complaint that turned out
to be valid, and it made me dig deeper and fix some generic non-
blocking problems I disovered in the code.
While cleaning this up, I moved session-specific stuff over to a
new session.h header from the libssh2_priv.h header.
By making 'data' and 'data_len' more local in several places in
this file it will be easier to spot how they are used and we'll
get less risks to accidentally do bad things with them.
clang-analyzer pointed out how 'data' could be accessed as a NULL
pointer if the wrong state was set, and while I don't see that
happen in real-life the code flow is easier to read and follow by
moving the LIBSSH2_FREE() call into the block that is supposed to
deal with the data pointer anyway.
clang-analyzer pointed out how 'data' could be accessed as a NULL
pointer if the wrong state was set, and while I don't see that
happen in real-life the code flow is easier to read and follow by
moving the LIBSSH2_FREE() call into the block that is supposed to
deal with the data pointer anyway.
clang-analyzer pointed this out as a "Pass-by-value argument in
function call is undefined" but while I can't see exactly how
this can ever happen in reality I think a little check for safety
isn't such a bad thing here.
The previously existing libssh2_scp_send_ex() function has no way
to send files that are larger than 'size_t' which on 32bit
systems mean 4GB. This new API uses a libssh2_int64_t type and
should thus on most modern systems be able to send enormous
files.
I'll introduce a new internal function set named
_libssh2_store_u32
_libssh2_store_u64
_libssh2_store_str
That can be used all through the library to build binary outgoing
packets. Using these instead of the current approach removes
hundreds of lines from the library while at the same time greatly
enhances readability. I've not yet fully converted everything to
use these functions.
I've converted LOTS of 'unsigned long' to 'size_t' where
data/string lengths are dealt with internally. This is The Right
Thing and it will help us make the transition to our
size_t-polished API later on as well.
I'm removing the PACKET_* error codes. They were originally
introduced as a set of separate error codes from the transport
layer, but having its own set of errors turned out to be very
awkward and they were then converted into a set of #defines that
simply maps them to the global libssh2 error codes instead. Now,
I'l take the next logical step and simply replace the PACKET_*
defines with the actual LIBSSH2_ERROR_* defines. It will increase
readability and decrease confusion.
I also separated packet stuff into its own packet.h header file.
We reserve ^libssh2_ for public symbols and we use _libssh2 as
prefix for internal ones. I fixed the intendation of all these
edits with emacs afterwards, which then changed it slightly more
than just _libssh2_error() expressions but I didn't see any
obvious problems.
This function no longer has any special purpose code for the
single entry case, as it was pointless.
The previous code would overflow the buffers with an off-by-one
in case the file name or longentry data fields received from the
server were exactly as long as the buffer provided to
libssh2_sftp_readdir_ex.
We now make sure that libssh2_sftp_readdir_ex() ALWAYS zero
terminate the buffers it fills in.
The function no longer calls the libssh2_* function again, but
properly uses the internal sftp_* instead.
When we ignore the EAGAIN from the transport layer within channel_write, we
now drain the outgoing transport layer buffer so that remainders in that
won't cause any problems in the next invoke of _libssh2_transport_write()
When sending data in a loop, we must not return EAGAIN if we
managed to send data in an earlier round. This was reported in
bug #126 => http://libssh2.stuge.se/ticket/126
When _libssh2_channel_write() is asked to send off 9 bytes, the
code needs to deal with the situation where less than 9 bytes
were sent off and prepare to send the remaining piece at a later
time.
As reported in bug report #166http://libssh2.stuge.se/ticket/166
by 'ptjm', the maximum window size must be less crazy for libssh2
to do better with more server implementations. I did not do any
testing to see how this changes raw SCP performance, but the
maximum window size is still almost 4MB. This also has the upside
that libssh2 will use less memory.
commit 7317edab61d2179febc38a2c2c4da0b951d74cbc cleared the outbound
blocking bit when send_existing() returned PACKET_NONE and *ret=0, as
opposed to before even calling send_existing(), but because *ret=1 when
sending parts 2..n of an existing packet, the bit would only be cleared
when calling libssh2_transport_write() for a new packet.
Clear the direction flag after the final part of a packet has been sent.
Suyog Jadhav pointed out that when receiving a window adjust to
a channel not found, the code would reference a NULL pointer.
Now it will instead output a message about that fact.
Comments in known_hosts file were not handle properly. They were parsed as
part of the key causing key matching to return a mismatch if the entry had a
comment. This adds a new API function that takes an optional comment and
changes libssh2_knownhost_readline to parse the comment as pass it to the
new function.
Fixes#164.
when _libssh2_packet_requirev() returns an error when waiting for
SSH_MSG_USERAUTH_SUCCESS or SSH_MSG_USERAUTH_FAILURE, it is an
error and it should be treated as such
libssh2_error() no longer allocates a string and only accepts a const
error string. I also made a lot of functions use the construct of
return libssh2_error(...) instead of having one call to
libssh2_error() and then a separate return call. In several of those
cases I then also changed the former -1 return code to a more
detailed one - something that I think will not change behaviors
anywhere but it's worth keeping an eye open for any such.
Sending SSH_MSG_CHANNEL_CLOSE without channel EOF is explicitly allowed
in RFC 4254, but some non-conforming servers will hang or time out when
the channel is closed before EOF.
Other common clients send and receive EOF before closing, there are no
drawbacks, and some servers need it to work correctly.
The libssh2 API calls should set the last error code and a message when
returning a failure by calling libssh2_error. This changeset adds these
calls to the libssh2_knownhost_* API as well as libssh2_base64_decode.
This change also makes libssh2_error into a function rather than a macro.
Its implementation is moved to misc.c. This function returns the error
code passed to it allowing callers to return the error value directly
without duplicating the error code.
When removing a known host, libssh2_knownhost_del would remove the node from the linked list, free its memory and then overwrite the struct parameter (which indicated which node to remove) with 0. However, this struct is actually allocated within the just-freed node meaning we're writing to freed memory. This made Windows very upset.
The fix is simply to overwrite the struct first before freeing the memory.
To make sure the public API is functional and that the
BLOCK_ADJUST_ERRNO() macro works correctly we MUST make sure to
call libssh2_error() when we return errors.
Mr anonymous in bug #125 pointed out that the userauth_keyboard_interactive()
function does in fact assign the same pointer a second time to a new allocated
buffer without properly freeing the previous one, which caused a memory leak.
To allow the libssh2_session_last_error() function to work as
documented, userauth_password() now better makes sure to call
libssh2_error() everywhere before it returns error.
Pointed out by mr anonymous in bug #128
all #defined macros in the public headers are considered to be part
of the API and I've generated individual man pages for each of them
to A) make it easier to figure out what each function/macro actually
is for so that automated lookups work better and for B) make sure we
have all public functions document (both macros and functions) to
make it easier for us to work away from all the macros in a future
release.
Fix memoary leak: if there was an "output" still allocated when a
session was torn down it needs to be freed in session_free()
Patch by Yoichi Iwaki in bug #2929647
Building libssh2-1.2.3 on Tru64 fails at line 48 and 166 because socklen_t
isn't defined on Tru64 unless _POSIX_PII_SOCKET is defined.
This patch updates configure.ac to add -D_POSIX_PII_SOCKET when building
on Tru64 platform(s).
Solaris builds of libssh2-1.2.3 failed on both x64 and UltraSPARC
platforms because of two problems:
1) src/agent.c:145 sun is a reserved word when using the SUNWspro compiler
2) example/direct_tcpip.c:84 INADDR_NONE is not defined
Neither libssh2_userauth_password_ex() nor
libssh2_userauth_keyboard_interactive_ex() would return a login failure
error if the server responded with a SSH_MSG_USERAUTH_FAILURE, instead
you would see whatever previous error had occurred, typically
LIBSSH2_ERROR_EAGAIN.
This patch changes error code -18 to LIBSSH2_ERROR_AUTHENTICATION_FAILED
and makes LIBSSH2_ERROR_PUBLICKEY_UNRECOGNIZED an alias for
LIBSSH2_ERROR_AUTHENTICATION_FAILED. In addition, new logic in
userauth_password() properly handles SSH_MSG_USERAUTH_FAILURE and both
this function and userauth_keyboard_interactive() now properly return
LIBSSH2_ERROR_AUTHENTICATION_FAILED.
The trace context is actually a bitmask so that tracing output can be
controlled by setting a bitmask using libssh2_trace(). However, the logic
in libssh2_debug() that converted the context to a string was using the
context value as an array index. Because the code used a bounds check on
the array, there was never a danger of a crash, but you would certainly
either get the wrong string, or "unknown".
This patch adds a lookup that iterates over the context strings and uses
it's index to check for the corresponding bit in the context.
The libssh2_trace_sethandler() call allows the user to handle the output of libssh2 rather than having it written to stderr. This patch updates libssh2_trace_sethandler() to allow a user-defined void* context value to be passed back to the output handler.
buildconf copies the template to example/ and configure makes sure
to generate a proper file from it and the direct_tcpip.c example
is the first one to use it - to make sure it builds fine on more
paltforms
userauth_publickey_fromfile() reads the key from a
file using file_read_publickey() which returns two
allocated strings, the decoded key and the key
method (such as "ssh-dss"). The latter can be
derived from the former but returning both avoids a
later allocation while doing so.
Older versions of userauth_publickey_fromfile() used
this method string directly but when
userauth_publickey() was factored out of
userauth_publickey_fromfile() it derived the method
from the key itself. This resulted in the method
being allocated twice.
This fix, which maintains the optimisation that
avoids an extra allocation, changes
userauth_publickey() so it doesn't allocate and
derive the method when userauth_pblc_method already
has a value.
Signed-off-by: Alexander Lamaison <awl03@doc.ic.ac.uk>
Commit 70b199f47659a74b8778c528beccf893843e5ecb introduced a parsing
bug in file_read_publickey() which made the algorithm name contain an
extra trailing space character, breaking all publickey authentication.
While this is code not currently in use, it is part of the generic linked
list code and since I found the error I thought I'd better fix it since we
might bring in this function into the code one day.
In case of failure we must make sure that the data we return
doesn't point to a memory area already freed. Reported anonymously
in the bug report #2910103.
Commit 683aa0f6b52fb1014873c961709102b5006372fc made send_existing() send
more than just the second part of a packet when the kernel did not accept
the full packet, but the function still overlooked the SSH protocol
overhead in each packet, often 48 bytes.
If only the last few bytes of a packet remained, then the packet would
erroneously be considered completely sent, and the next call to write
more data in the session would return a -39 error.
DSA signatures consist of two 160-bit integers called r and s. In ssh-dss
signature blobs r and s are stored directly after each other in binary
representation, making up a 320-bit (40 byte) string. (See RFC4253 p14.)
The crypto wrappers in libssh2 would either pack r and s incorrectly, or
fail, when at least one integer was small enough to be stored in 19 bytes
or less.
The patch ensures that r and s are always stored as two 160 bit numbers.
When libssh2_transport_write() is called to continue sending a
partially sent packet the write direction flag must not be cleared
until the previous packet has been completely sent, or the app would
hang if the packet still isn't sent completely, since select() gets
called by the internal blocking emulation layer in libssh2 but would
then not be watching the socket for writability.
Clear the flag only once processing of previous packet data is
complete and a new packet is about to be prepared.
The forward accepting was not done right before, and the
packet_queue_listener function didn't assign a necessary
variable. All fixed by Juzna. I (Daniel) modified the
forward_accept() change somewhat.
First, win32/msvcproj.{head,foot} are now committed with CRLF line endings,
and .gitattributes specifies that these should not be changed on checkout or
commit. These are win32 files so it makes sense to store them with native
line endings.
Second, the rules for generating libssh2.dsp and libssh2.vcproj are changed
so that the full file contents passes through awk, which strips all CR and
then prints each line with one CRLF line ending. Stripping CR is important
to avoid CRCRLF in case the input already comes with CRLF.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
