The default channel window size used until now was 256KB. This value is
too small and results on a bottleneck on real-life networks where
round-trip delays can easily reach 300ms.
The issue was not visible because the configured channel window size
was being ignored and a hard-coded value of ~22MB being used instead,
but that was fixed on a previous commit.
This patch just changes the default window size
(LIBSSH2_CHANNEL_WINDOW_DEFAULT) to 2MB. It is the same value used by
OpenSSH and in our opinion represents a good compromise between memory
used and transfer speed.
Performance tests were run to determine the optimum value. The details
and related discussion are available from the following thread on the
libssh2 mailing-list:
http://www.libssh2.org/mail/libssh2-devel-archive-2013-10/0018.shtmlhttp://article.gmane.org/gmane.network.ssh.libssh2.devel/6543
An excerpt follows:
"I have been running some transfer test and measuring their speed.
My setup was composed of a quad-core Linux machine running Ubuntu 13.10
x86_64 with a LXC container inside. The data transfers were performed
from the container to the host (never crossing through a physical
network device).
Network delays were simulated using the tc tool. And ping was used to
verify that they worked as intended during the tests.
The operation performed was the equivalent to the following ssh command:
$ ssh container "dd bs=16K count=8K if=/dev/zero" >/dev/null
Though, establishment and closing of the SSH connection was excluded
from the timings.
I run the tests several times transferring files of sizes up to 128MB
and the results were consistent between runs.
The results corresponding to the 128MB transfer are available here:
https://docs.google.com/spreadsheet/ccc?key=0Ao1yRmX6PQQzdG5wSFlrZl9HRWNET3ZyN0hnaGo5ZFE&usp=sharing
It clearly shows that 256KB is too small as the default window size.
Moving to a 512MB generates a great improvement and after the 1MB mark
the returns rapidly diminish. Other factors (TCP window size, probably)
become more limiting than the channel window size
For comparison I also performed the same transfers using OpenSSH. Its
speed is usually on par with that of libssh2 using a window size of 1MB
(even if it uses a 2MB window, maybe it is less aggressive sending the
window adjust msgs)."
Signed-off-by: Salvador Fandino <sfandino@yahoo.com>
Store but don't use keys of unsupported types on the known_hosts file.
Currently, when libssh2 parses a known_host file containing keys of some
type it doesn't natively support, it stops reading the file and returns
an error.
That means, that the known_host file can not be safely shared with other
software supporting other key types (i.e. OpenSSH).
This patch adds support for handling keys of unknown type. It can read
and write them, even if they are never going to be matched.
At the source level the patch does the following things:
- add a new unknown key type LIBSSH2_KNOWNHOST_KEY_UNKNOWN
- add a new slot (key_type_name) on the known_host struct that is
used to store the key type in ascii form when it is not supported
- parse correctly known_hosts entries with unknown key types and
populate the key_type_name slot
- print correctly known_hosts entries of unknown type
- when checking a host key ignore keys that do not match the key
Fixes#276
The new libssh2_sftp_fsync API causes data and metadata in the
currently open file to be committed to disk at the server.
This is an OpenSSH extension to the SFTP protocol. See:
https://bugzilla.mindrot.org/show_bug.cgi?id=1798
... in macro parameters to avoid compiler warnings about lost precision.
Several macros in libssh2.h call strlen and pass the result directly to
unsigned int parameters of other functions, which warns about precision
loss because strlen returns size_t which is unsigned long on at least
some platforms (such as OS X). The fix is to simply typecast the
strlen() result to unsigned int.
libssh2_knownhost_readfile() silently ignored problems when reading keys
in unsupported formats from the known hosts file. When the file is
written again from the internal structures of libssh2 it gets truntcated
to the point where the first unknown key was located.
* src/knownhost.c:libssh2_knownhost_readfile() - return error if key
parsing fails
INVALID_SOCKET is a special value in Windows representing a
non-valid socket identifier. We were #defining this to -1 on
non-Windows platforms, causing unneccessary namespace pollution.
Let's have our own identifier instead.
Thanks to Matt Lawson for pointing this out.
Some SFTP servers send SFTP packets larger than 40000. Since the limit
is only present to avoid insane sizes anyway, we can easily bump it.
The define was formerly in the public header libssh2_sftp.h but served
no external purpose and was moved into the source dir.
Bug: http://www.libssh2.org/mail/libssh2-devel-archive-2011-11/0004.shtml
Reported by: Michael Harris
When cross compiling to Windows, libssh2.h include Windows header files
with upper case filenames : BaseTsd.h and WinSock2.h.
These files have lowercase names with mingw-w64 (iirc, it's the same with
mingw). And as on Windows, being lowercase or uppercase does not matter.
As discussed on the mailing list, it was wrong for win64 and using the
VC-provided type is the safest approach instead of second- guessing
which one it should be.
We now allow libssh2_session_flag() to enable compression with a new
flag and I added documentation for the previous LIBSSH2_FLAG_SIGPIPE
flag which I wasn't really aware of!
The function libssh2_session_startup() is now considered deprecated due
to the portability issue with the socket argument.
libssh2_session_handshake() is the name of the replacement.
No idea why we had this ifdef at all but MSVC, MingW32, Watcom
and Borland all have no sys/uio.h header; so if there's another
Win32 compiler which needs it then it should be added explicitely
instead of this negative list.
The condition around the ssize_t typedef depended on both LIBSSH2_WIN32
*and* _MSC_VER being defined when it should be enough to depend on
_MSC_VER only. It also makes it nicer so libssh2-using code builds fine
without having custom defines.
As the long-term goal is to get rid of the extensive set of
macros from the API we can just as well start small by not adding
new macros when we add new functions. Therefore we let the
function be libssh2_sftp_statvfs() plainly without using an _ex
suffix.
I also made it use size_t instead of unsigned int for the string
length as that too is a long-term goal for the API.
OpenSSH has ways to add hosts to the knownhosts file that include
a specific port number which makes the key associated with only
that specific host+port pair. libssh2 previously did not support
this, and I was forced to add a new function to the API to
properly expose this ability to applications:
libssh2_knownhost_checkp()
To *add* such hosts to the knownhosts file, you make sure to pass
on the host name in that manner to the libssh2_knownhost_addc()
function.
The previously existing libssh2_scp_send_ex() function has no way
to send files that are larger than 'size_t' which on 32bit
systems mean 4GB. This new API uses a libssh2_int64_t type and
should thus on most modern systems be able to send enormous
files.
