Frank Heckenbach
2014-11-26 05:31:18 UTC
Hi everybody,
this is to announce my plan to port the Etherlab code to userspace.
I'll explain my reasons and roadmap below. If someone is interested
in this or has some comments, please let me know. Otherwise, I
expect to proceed on my own and publish the result on my web site
when finished.
Reasons:
- I had to make a number of changes to the Etherlab code to fix some
bugs and make it useable to us. The Etherlab developers are
obviously not interested in those changes, so I have to maintain
them myself. As was discussed on this list some months ago,
keeping up with newer Etherlab versions would cost me additional
maintenance and testing effort (already now since my changes are
based on 1.5.0, whereas 1.5.2 has some conflicting changes), so
I'd use my own fork of 1.5.0 which is known to work for us rather
than 1.5.2 anyway.
- Keeping up with new kernel versions is also not always easy
(especially for the drivers which are patched files from the
standard kernel, but also other kernel interfaces are known to
change often), whereas userspace code is much easier to maintain
(incompatible library changes are quite rare).
- So far we've been using RTAI for our realtime code. But that's
also always been a bit troublesome (kernel version dependencies,
high crash potential in case of problems, additional code with its
own set of bugs, etc.), so we'd rather try to get rid of it
anyway. Meanwhile the RT capabilities of the standard kernel have
improved in recent years, and due to the wide availability of SMP,
we can, if necessary, increase RT-ability by using CPU affinity
(reserve one CPU for RT code, leave the other CPUs for the rest --
of course, kernel code, esp. network drivers might need special
consideration here).
- Our application code uses a lot of floating point which is
supported in RTAI (though with some quirks), not in non-RTAI
kernel mode, but of course easily in userspace.
- Userspace code is generally much easier to debug.
- If I had known and considered all this back then, I might have
started with other code instead of Etherlab which is already
userspace based, but has different interfaces (and possibly
different bugs). But as things are now, since my code is tightly
bound to the Etherlab interface, and well tested with (the patched
version of) it, it seems easier to port this code to userspace
than change my application code.
Roadmap:
- Start with 1.5.0 plus my patches.
- Replace kernel infrastructure by corresponding userspace
functionality (e.g. kthread -> pthread, semaphore ->
pthread_mutex, kmalloc -> malloc, kprintf -> fprintf); copy kernel
utilities that have no actual kernel dependencies (in particular
kernel lists).
- Turn the code into a library which the application can use, so the
userspace application thread(s) will do what the (normal or RTAI)
kernel application thread(s) did and, being part of the same
process, they can access the master's data structures (e.g. PDO
memory, datagram queues) without explicit shared memory or such.
(Though the library will start some threads of its own, e.g.
master, EoE and cdev (see below), just like the master now starts
some kthreads.) This will not allow the master to stay resident
while the application is restarted, so it will need to re-scan the
bus every time. This is not a high-priority problem for me
(because we rarely restart our application code, and when we do,
the hardware reinitialization takes longer than the bus scan
anyway). If needed, one could solve this problem by putting a
layer in userspace in between which would then need to do IPC
and/or use shared memory for communication.
- Use the generic backend driver which also shouldn't require too
many changes since it already uses a raw ethernet socket which is
also available from userspace).
- Implement EoE via a tap/tun device.
- Replace the cdev ioctl interface by a (normal TCP) socket and
adapt the library to use it. Basically wrap each ioctl data
structure as-is together with its cmd value in a data packet.
- Omit the mmap for the cdev because I don't need it. (If someone
has a need for it, it could be supported via userspace shared
memory.)
- Omit the tty because I don't need it. (If someone has a need for
it, it could be supported via a pty.)
- Generally, make as few changes to the code as possible
(e.g. #define stuff in a common header rather than making global
replacements wherever possible).
Regards,
Frank
this is to announce my plan to port the Etherlab code to userspace.
I'll explain my reasons and roadmap below. If someone is interested
in this or has some comments, please let me know. Otherwise, I
expect to proceed on my own and publish the result on my web site
when finished.
Reasons:
- I had to make a number of changes to the Etherlab code to fix some
bugs and make it useable to us. The Etherlab developers are
obviously not interested in those changes, so I have to maintain
them myself. As was discussed on this list some months ago,
keeping up with newer Etherlab versions would cost me additional
maintenance and testing effort (already now since my changes are
based on 1.5.0, whereas 1.5.2 has some conflicting changes), so
I'd use my own fork of 1.5.0 which is known to work for us rather
than 1.5.2 anyway.
- Keeping up with new kernel versions is also not always easy
(especially for the drivers which are patched files from the
standard kernel, but also other kernel interfaces are known to
change often), whereas userspace code is much easier to maintain
(incompatible library changes are quite rare).
- So far we've been using RTAI for our realtime code. But that's
also always been a bit troublesome (kernel version dependencies,
high crash potential in case of problems, additional code with its
own set of bugs, etc.), so we'd rather try to get rid of it
anyway. Meanwhile the RT capabilities of the standard kernel have
improved in recent years, and due to the wide availability of SMP,
we can, if necessary, increase RT-ability by using CPU affinity
(reserve one CPU for RT code, leave the other CPUs for the rest --
of course, kernel code, esp. network drivers might need special
consideration here).
- Our application code uses a lot of floating point which is
supported in RTAI (though with some quirks), not in non-RTAI
kernel mode, but of course easily in userspace.
- Userspace code is generally much easier to debug.
- If I had known and considered all this back then, I might have
started with other code instead of Etherlab which is already
userspace based, but has different interfaces (and possibly
different bugs). But as things are now, since my code is tightly
bound to the Etherlab interface, and well tested with (the patched
version of) it, it seems easier to port this code to userspace
than change my application code.
Roadmap:
- Start with 1.5.0 plus my patches.
- Replace kernel infrastructure by corresponding userspace
functionality (e.g. kthread -> pthread, semaphore ->
pthread_mutex, kmalloc -> malloc, kprintf -> fprintf); copy kernel
utilities that have no actual kernel dependencies (in particular
kernel lists).
- Turn the code into a library which the application can use, so the
userspace application thread(s) will do what the (normal or RTAI)
kernel application thread(s) did and, being part of the same
process, they can access the master's data structures (e.g. PDO
memory, datagram queues) without explicit shared memory or such.
(Though the library will start some threads of its own, e.g.
master, EoE and cdev (see below), just like the master now starts
some kthreads.) This will not allow the master to stay resident
while the application is restarted, so it will need to re-scan the
bus every time. This is not a high-priority problem for me
(because we rarely restart our application code, and when we do,
the hardware reinitialization takes longer than the bus scan
anyway). If needed, one could solve this problem by putting a
layer in userspace in between which would then need to do IPC
and/or use shared memory for communication.
- Use the generic backend driver which also shouldn't require too
many changes since it already uses a raw ethernet socket which is
also available from userspace).
- Implement EoE via a tap/tun device.
- Replace the cdev ioctl interface by a (normal TCP) socket and
adapt the library to use it. Basically wrap each ioctl data
structure as-is together with its cmd value in a data packet.
- Omit the mmap for the cdev because I don't need it. (If someone
has a need for it, it could be supported via userspace shared
memory.)
- Omit the tty because I don't need it. (If someone has a need for
it, it could be supported via a pty.)
- Generally, make as few changes to the code as possible
(e.g. #define stuff in a common header rather than making global
replacements wherever possible).
Regards,
Frank