KNZL: Seb Kienzl's page

SDRAM problems with STM32F427/437/429/439

2017-12-13T00:00:00+00:00

The FMC (flexible memory controller) of these microcontrollers is a bug ridden beast. The latest errata sheet lists a total of 15 “silicon limitations”. Two of them are connected to SDRAM and are particularily ugly, as they may result in corrupt reads. As of now (Dec 2017), both of them are not fixed in any revision, so you may want to continue reading if you use any of these types with SDRAM…

A mysterious bug…

The last few days, my client and I were hunting a mysterious bug: When the STM32F439 was really busy (i.e. lots of interrupts), very, very seldomly two invalid bytes would arrive at the host over USB.

After putting checks in place and making sure the data gets handed to the USB-stack correctly, it turned out that when the bug occured, the invalid data was actually read from the SDRAM, although the same data seemed ok there.

After re-checking that the SDRAM timing and initialization was correct, studying the latest errata sheet revealed a likely candidate: “2.11.15 SDRAM bank address corruption upon an interruption of CPU read burst access”.

If the CPU gets an interrupt while it is busy with an LDM from SDRAM, the next read from another bank of the SDRAM may result in wrong data read. Öha.

Who should be concerned?

Everyone that uses a STM32F427/STM32F437/STM32F429/STM32F439 with SDRAM. Here, the problem went unnoticed for a couple of years, although these microcontrollers have been used in numerous projects. Only in one project under very specific circumstances the problem surfaced.

The workarounds

Luckily, there are several workarounds for every taste.

Workaround #1: 32-bit SDRAM

If you’re using 32-bit SDRAM, you could simply disable the SDRAM’s FIFO by clearing RBURST in FMC_SDCR1/FMC_SDCR2 when initialzing the FMC.

Why only 32-bit SDRAM? Because there’s another bug in the FMC: “2.11.5 Interruption of CPU read burst access to an end of SDRAM row” that only affects 8- and 16-bit SDRAMs. The workaround for this problem would be turning the FIFO on, which is the opposite of the workaround for the problem we’re trying to solve.

But anyway, turning off the FIFO is a viable workaround when using 32-bit SDRAMs. It will probably hurt SDRAM-performance a little, but maybe that’s not an issue in your case.

Workaround #2: Set DISMCYCINT

Another workaround I came up with that’s not mentioned in the errata sheet is to set DISMCYCINT in the “Auxiliary control register” (ACTLR). If this bit is set, LDMs will not get interrupted and the problem will go away.

There’s a drawback, though: The worst-case interrupt latency increases.

This FAQ entry tells us that the longest possible burst is a VLDM Rx, [S0-S31], which is 32 words or 128 bytes.

In our configuration (16-bit SDRAM @ 90 MHz), such a burst takes almost 1 µs. This may be too long if you have really hard realtime requirements. Other than that, it’s a very simple and effective workaround.

Workaround #3: “Precharge all” before the 2nd read

Another workaround mentioned in the errata sheet is to “[…] send a PRECHARGE ALL command before the second read access […]”.

As before the second read access is after the first (interrupted) access, we could send a “precharge all” command first thing in every ISR.

To achieve that, one can wrap all ISRs with a specially crafted vector table.

A few things to consider:

VTOR must point to this table.
Any vector-table must be aligned to 128 words (512 bytes).
To be on the safe side, make sure the first two entries (initial SP and initial PC) contain proper values. There may be some code using them for a reset.

Then, have all interrupt-vectors point to a routine something like that:

isr_wrapper
	; send PRECHARGE ALL command to SDRAM 1 via FMC
	LDR    R0, =0xa0000150 ; FMC_SDCMR register
	MOVS   R1, #0x12       ; Send PALL to SDRAM 1
	STR    R1, [R0]
	
	MRS    R0, IPSR        ; Get current interrupt number ..
	; .. and use it as the index to look up the actual ISR and branch there
	; (in our case, we are using a vector table in RAM)
	LDR    R1, =__VECTOR_RAM
	LDR    R0, [R1, R0, LSL #2]
	BX     R0

If you have two SDRAMs, you’ll have to send PALL to both of them. Also, your actual vector table is probably called differently, or it may just sit at address 0 if you’re using the vector table in Flash ROM.

This is the workaround we finally went with as it hardly increases interrupt latency and is not too invasive. Long-term testing showed that with this in place, the problem doesn’t occur any more.

systemd-logind deletes your message queues (RemoveIPC)

2017-08-05T00:00:00+00:00

When something unexpected happens in a GNU/Linux system nowadays, it’s likely that systemd is involved somehow. If you want yet another spectacular example of how systemd and especially its author know what’s best for everyone, read on…

I wrote software for GNU/Linux where two processes communicate through a POSIX message queue (see man mq_overview or man mq_open).

So far so good, except that the message queue kept on disappearing after a while for no appearant reason.

I found that systemd mounts the mqueue-pseudo-filesystem at /dev/mqueue, which contains all message queues. It seemed quite likely that the queue was repeatedly being deleted from there.

A quickly written jprobe on mqueue_unlink revealed the culprit: systemd-logind.

Once again my gut-feeling that systemd is involved was right. Now if you (like I did) think this is a bug: It isn’t.

After reading the source and finding out that it’s actually default behaviour (see this commit) to remove “POSIX IPC Objects” when a user “fully logs out” (see bottom of man logind.conf) I wasn’t very surprised about You-Know-Who telling us that it’s the right thing to do by default.

My advise: Put RemoveIPC=no into your /etc/systemd/logind.conf or better avoid systemd altogether if you can. Here’s a collection of more systemd-crazyness.

SoundFX Macs Opera support for AdPlug

2017-01-13T00:00:00+00:00

Many of you remember the FM-sounds that the OPL-chips on old soundcards like the AdLib or the Sound Blaster make. One cold day last December, I remembered that when I was around 12 years old, I bought tracker-software for creating AdLib-tunes. More important things had to wait, I had an “open invoice” with that specific piece of software…

It was boxed software on one 5¼” disk and I bought it because the box promised that with that software, I’d be able to add music to my own programs!

Being an avid QuickBASIC programmer (and electric guitar player) at that age, I was hooked. However, my enthusiasm waned off when I read the documentation on the TSR that came with it: Allocating memory? Calling an interrupt? What?

None of my computer-friends could tell me how this works. Since this was one of the few softwares that I actually bought back then, I was disappointed.

Some 25 years later I didn’t remember the name of the software any more, but after some looking around I stumbled over YouTube videos by a Mr. Grzondziel that showed the tracker. Thanks for sending me the software! Oddly enough, I was in contact with him a few years before that about some other topic.

Also, I noticed that AdPlug, an open source AdLib player, didn’t have support for files composed with this tracker.

I couldn’t find the original author, so after some reverse engineering (and running Turbo Debugger through a virtual serial port on two DOSBoxes) and lots of listening, here goes: SoundFX Macs Opera support for AdPlug.

And this is how it sounds, this song is “Necronom” by Michael Tschögl:

CMake: EXCLUDE_FROM_ALL vs. EXCLUDE_FROM_DEFAULT_BUILD

2016-11-02T00:00:00+00:00

add_library(target EXCLUDE_FROM_ALL ...) is really convenient with Make-generators if you want to define libraries that should only be built if another target depends on them. This currently doesn’t work (well) for Visual Studio generators, but here’s a trick how to get this behaviour in Visual Studio as well…

(Written for CMake 3.6)

EXCLUDE_FROM_ALL

As written in the CMake docs, the EXCLUDE_FROM_ALL target-property causes a target to be excluded from the “all” build-target, which is the default if make is started without any args.

If used for static libraries (e.g. by using add_library(target EXCLUDE_FROM_ALL ...)), the library will be built nevertheless if another target (library or executable) depends on it.

Like this, you can define a bunch of libraries (e.g. in your shared codebase) and only those that are actually needed will be built.

This reduces build times and makes it easier to handle component-dependencies.

I’m not sure if this use case is what the implementor of EXCLUDE_FROM_ALL had in mind, but I find it practical like that.

EXCLUDE_FROM_DEFAULT_BUILD

Interestingly, there’s the EXCLUDE_FROM_DEFAULT_BUILD target property which on first sight seems to be somewhat redundant.

But it isn’t, it’s a property specific to the Visual Studio generator and only says that a target (i.e. a Visual Studio project) isn’t built when the Solution is build by pressing “Build Solution (F7)”.

The difference to EXCLUDE_FROM_ALL: If a target is excluded from build like that, it won’t be built even if another target depends on it.

This doesn’t mean that EXCLUDE_FROM_ALL is ineffective with Visual Studio: If you build the “ALL_BUILD” project, you’ll get the same behaviour as with Make.

While this works, I don’t like it: People are used to hitting F7 (or whatever the shortcut is) and the project should build.

The solution

My use case:

A bunch of static libraries in the shared codebase where not every library is needed in every project
A (CMake-) project consists of one or more executables
Builds with Visual Studio and Make

Others seem to have wanted to do the same:

This is my solution for this problem:

# Override add_library() and set the "EXCLUDE_FROM_DEFAULT_BUILD" property to TRUE on all libraries that use "EXCLUDE_FROM_ALL".
function(add_library TARGET)
	_add_library(${TARGET} ${ARGN})
	cmake_parse_arguments(OPTS "EXCLUDE_FROM_ALL" "" "" ${ARGN})
	if(OPTS_EXCLUDE_FROM_ALL)
		set_target_properties(${TARGET} PROPERTIES EXCLUDE_FROM_DEFAULT_BUILD TRUE)
	endif()
endfunction()

# Helper function to recursively set EXCLUDE_FROM_DEFAULT_BUILD to FALSE on all targets "TARGET" depends on
function(_include_in_build TARGET)
	if(TARGET ${TARGET})
		get_target_property(TYPE ${TARGET} TYPE)
		if(TYPE MATCHES "INTERFACE")
			get_target_property(DEPS ${TARGET} INTERFACE_LINK_LIBRARIES)
		else()
			set_target_properties(${TARGET} PROPERTIES EXCLUDE_FROM_DEFAULT_BUILD FALSE)
			get_target_property(DEPS ${TARGET} LINK_LIBRARIES)
		endif()
		if(DEPS)
			foreach(D ${DEPS})
				_include_in_build(${D})
			endforeach()
		endif()
	endif()
endfunction()

# Take executables as roots to run _include_in_build() on
function(target_link_libraries TARGET)
	_target_link_libraries(${TARGET} ${ARGN})
	
	get_target_property(_TYPE ${TARGET} TYPE)
	if(_TYPE MATCHES "EXECUTABLE")
		_include_in_build(${TARGET})
	endif()
endfunction()

How does it work?

First, a trick I like to use: Functions like add_library and target_link_libraries can be “overridden” by just defining a new version. The original function is still available as e.g. _add_library or _target_link_libraries.
For every library that is defined which has the EXCLUDE_FROM_ALL property set the EXCLUDE_FROM_DEFAULT_BUILD property is set as well.
For every executable target on which target_link_libraries() is used, the function _include_in_build() recursively sets EXCLUDE_FROM_DEFAULT_BUILD to FALSE on all targets the executable depends on. Imagine the executable as the root of a tree of dependencies which is traversed.
There’s special handling for “interface targets”, as they don’t have the EXCLUDE_FROM_DEFAULT_BUILD property.

Conclusion

While this works for my use case, I think a better way would be to have the ability to define “non-standalone” targets in CMake. Targets like this would not be built if no other target depends on them. Something to be discussed with the CMake devs…

That’s all, thanks for reading!

Debugging IAR ELFs with GDB on the STM32F4xx

2016-09-14T00:00:00+00:00

There are situations where you may not want to use IAR’s C-SPY for debugging but rather would go with GDB.

So you tell yourself “an ELF is an ELF”, start up J-Link GDB server, attach GDB to it, load the ELF, continue and… nothing.

There are several things that can go wrong, read on to find out which…

Why use GDB instead of C-SPY?

My client uses IAR’s toolchain.

However, C-SPY currently (Sep 2016) has one major flaw: It takes forever to load big ELF files. Support told us they’ll improve it. GDB loads it instantly.

Also, you may want to run the ELF through a debugger in environments where you don’t have an IAR license.

My setup: Segger J-Link, J-Link GDB Server and gdb from the GNU ARM Embedded Toolchain.

What’s the problem, then? Maybe it’s one of the following reasons.

Reason #1: Program entry position

GDB honours the “Program entry position” from the ELF header and sets the program counter (PC) accordingly on load.

In my case the IAR linker sets the program entry position to the address of __iar_program_start. It is an entry position, sort of, but really it isn’t. Rather, it should be something like the address of Reset_Handler.

Except the PC, also the initial stack pointer (SP) may be bogus.

To understand what PC/SP really should be, this is what happens when an STM32F4xx starts:

Set SP to the word stored at VTOR+0
Set PC to the word stored at VTOR+4
Start

What’s VTOR? It’s the “Vector table offset register”. After reset, it’s 0.

Is there any memory at 0x00000000? Yes there is, for usual configurations (i.e. boot from flash), the flash memory is aliased to address 0x00000000.

So, to get the proper values for PC/SP, I’ve found that resetting the CPU after the ELF has been loaded works well.

To illustrate this, look at this GDB session output:

C:\Users\seb\...>arm-none-eabi-gdb.exe test.elf
GNU gdb (GNU Tools for ARM Embedded Processors) 7.10.1.20160210-cvs
...
Reading symbols from test.elf...done.
(gdb) target remote localhost:2331
Remote debugging using localhost:2331
0x08000e20 in ?? ()
(gdb) monitor reset
Resetting target
(gdb) load
Loading section A1 rw, size 0x410 lma 0x8000000
Loading section P2 rw, size 0x3c lma 0x8000410
Loading section P3 ro, size 0x1059f8 lma 0x800044c
Start address 0x8043ec8, load size 1072708
Transfer rate: 33792 KB/sec, 4063 bytes/write.
(gdb) stepi
0x08043eca in __iar_program_start ()

That’s not the right start address, it should be:

(gdb) x/x 0x00000004
0x4:    0x08000fd1

After a reset, it will fetch the correct PC:

(gdb) monitor reset
Resetting target
(gdb) stepi
0x08000fd2 in ?? ()

Then, you can just continue the program.

So something like the following in your gdbinit should work:

# Connect to GDB Server
target remote localhost:2331

# Reset CPU
monitor reset

# Load ELF
load

# Reset again
monitor reset

Reason #2: Boot loader configuration

Now, it still may not work. For example, we use a boot loader configuration: Our boot loader sits in the first 256k of the flash and the actual application comes after these 256k. The application has its own vector table at 0x08040000.

Still, for some magical reason, C-SPY is perfectly fine loading and running this application and it uses the proper PC and SP from the vector table at 0x08040000. How does it do that?

I stumbled over a comment in startup_stm32f429_439xx.s:

The name "__vector_table" has special meaning for C-SPY: it is where the SP start value is found, and the NVIC vector table register (VTOR) is initialized to this address if != 0.

So this is what C-SPY does:

Set VTOR to the address of the symbol __vector_table
Fetch and set SP and PC from there
Run

Of course GDB doesn’t know about this, that’s why it has to be done manually:

# Adjust VTOR (Vector table offset register)
set {int}0xE000ED08 = __vector_table
# Set SP/PC to the values from the actual vector table
set $sp = *(int*)__vector_table
set $pc = *((int*)__vector_table+1)

0xE000ED08 is the address of the VTOR on the STM32F439 we’re using. I haven’t checked, but it’s probably the same for other controllers of this family.

That’s all, thanks for reading!

Speeding up DHCP on bad WiFi links

2016-06-05T00:00:00+00:00

In my current project I noticed that on bad WiFi links initial DHCP negotiations may take a really long time.

The reason is simple: In the initial steps, DHCP uses UDP broadcast packages that may get lost.

To find out more and learn how to speed things up, read on …

A short 802.11 primer

In embedded devices that use WiFi modules that only have egyptian PCB traces as antennas, the packet loss between the access point (AP) and the WiFi client (STA(tion) in 802.11 terminology) can be high. When receiving, I could provoke >50% lost packets using modules from different vendors with APs that would receive much better with a proper antenna. Also, the embedded device may operate in a noisy (RF wise) environment.

Two facts about 802.11 WLANs which I didn’t know before:

Packets are normally sent through the AP, event if two STAs in the same network (BSS) communicate with each other. There is (T)DLS “Direct Link Setup” in 802.11e/z, but it’s not relevant in this case.
MAC unicast packets (i.e. AP->STA or STA->AP) are ACK’d and are retransmitted a couple of times as needed.

By “MAC unicast” I mean that they’re unicast on the data link layer (2), i.e. the packets contain a destination MAC address of a specific device.

There are also “MAC broadcast” packets, which are only sent by the AP. These packets are not ACK’d. It wouldn’t make much sense to wait for all STAs to ACK a broadcast packet.

How does this play together with IP?

	IP unicast	IP broadcast
STA->AP	MAC unicast	MAC unicast
AP->STA(s)	MAC unicast	MAC broadcast

As you can see, IP broadcast packets from AP->STAtions are sent as MAC broadcast, which are not ACK’d and thus are more likely to get lost.

And this is exactly the problem.

(Side note: To measure packet loss, send UDP broadcast packets from Ethernet->AP->STA and count how many are lost.)

A short DHCP primer

These are the initial steps of DHCP:

Client -> Broadcast: DHCPDISCOVER (“I need an IP”)
Server -> Broadcast: DHCPOFFER (“Here’s one”)
Client -> Broadcast: DHCPREQUEST (“I’ll take it”)
Server -> Broadcast: DHCPACK (“OK”)

These are all single UDP packets, targeted to the broadcast addresses of both link (e.g. MAC FF:FF:FF:FF:FF:FF) and network layer (e.g. IPv4 255.255.255.255). Except packets sent by the STA, they have the AP’s MAC address as the destination address at the link layer.

With the explanations before, we now know that OFFER and ACK are the packets which are likely to get lost, because they’re not ACK’d.

To make matters worse, RFC 2131 dictates exponential backoff for client-retransmits (e.g. 4s/8s/16s/…) if no reponse is received for DISCOVER or REQUEST.

This should explain why initial (“cold boot”) DHCP negotiations may take a really long time.

How to speed things up

Use proper antennas

There are certified module/external antenna combinations. If you can, use them. Not only DHCP will work better, but the overall performance will be much better.

Use a proper DHCP client implementation

… or use your DHCP client properly.

What do I mean by that? What I’ve explained about DHCP is only the “cold boot” behaviour as you may know. If you look at Figure 5 of RFC 2131, there are many more states in a DHCP client’s life. So if the device would persist its DHCP information and timer “T1” wouldn’t have expired, on reboot the DHCP client could simply go through INIT_REBOOT->REBOOTING->BOUND with unicast packets.

Use my BROADCAST bit trick

In every packet sent by the DHCP client, there’s a “BROADCAST bit”.

This is what RFC 2131 (4.1) says about the BROADCAST bit:

A client that cannot receive unicast IP datagrams until its protocol software has been configured with an IP address SHOULD set the BROADCAST bit in the 'flags' field to 1 in any DHCPDISCOVER or DHCPREQUEST messages that client sends. The BROADCAST bit will provide a hint to the DHCP server and BOOTP relay agent to broadcast any messages to the client on the client's subnet. A client that can receive unicast IP datagrams before its protocol software has been configured SHOULD clear the BROADCAST bit to 0. The BOOTP clarifications document discusses the ramifications of the use of the BROADCAST bit [21].

So if the BROADCAST bit is set, the DHCP server will answer with a brodcast, if it’s cleared, it will answer unicast. When sending OFFER, it’ll send to the IP address it offers.

(Side note: The mentioned “BOOTP clarifications document” states the same.)

If the DHCP client can be modified to send DISCOVER/REQUEST with a cleared BROADCAST bit and if the rest of the TCP/IP stack can be made to accept unicast DHCP packets even if it hasn’t configured its IP yet, things will improve.

In my case, I’ve changed the DHCP client (it’s was only a tiny change) so that the first packet of DISCOVER/REQUEST has the BROADCAST bit cleared. Retransmits have it set to give bad DHCP server implementations a chance to answer.

Thanks for reading!

Update Feb 2021: There used to be an example here of how to have Segger’s embOS/IP behave as described. I just went through the Changelog and noticed that in V3.30 this behaviour has been implemented. See IP_DHCPC_ConfigUniBcStartMode().

Using byacc+Ragel for memory efficient parsers

2014-07-01T00:00:00+00:00

Parser and lexer generators are great tools and there is little reason to knit parsers or lexers (aka tokenizers) yourself.

If your (embedded) target is short on memory, the used tools have to be picked carefully, as not many of them are designed with memory-efficiency in mind.

I’ve found byacc and Ragel to be a great combination with both of them generating extremely tight code.

Want to know more about them and other tools? Read on!

These are the requirements:

Generated code depends on little or no runtime
The grammar should be usable for generation to other languages, e.g. Java or C# with little effort
Memory-overflow conditions should be detected and handled gracefully
Generated code should use as little RAM as possible, possibly statically allocated

There’s a table of available tools on Wikipedia.

The parser generator: byacc

Byacc (“Berkeley Yacc”) is a really nice Yacc-implementation, which generates an LALR-parser. It doesn’t have all the features that GNU bison (another Yacc variant) offers but it does produce nicer and tighter code. No wonder, as it seems that both bison and byacc have initially been written by the same person (a Robert Corbett), with byacc having been written after bison.

Byacc basically generates a few (ROMable) tables, optional debug-code and the parsing-logic. All of its dynamic allocations happen in one function called “yygrowstack”, which could be easily changed to use static memory only.

Like bison, it can generate a “pure” (reentrant) parser and can be given context, so the same parser can have multiple instances.

One thing isn’t ideal, which is that the initial stack size (YYINITSTACKSIZE) can’t be configured. That’s a pity, because in my case an initial stack size of 32 (vs. the default 500) is enough in all relevant cases. The stack isn’t the execution stack, but the stack where the shift/reduce operations are being performed on. So this isn’t bytes, but the number of tokens on the stack.

Talking about the token stack: You should know that If you’re freeing resources connected to a token (e.g. free() on a pointer in the YYSTYPE-union) in the action code then this code may not be called on parse errors.

To change YYINITSTACKSIZE, byacc’s skeleton.c can be patched or alternatively the generated file can be changed with sed or CMake’s file/string.

This change fulfills Req #4. Stack-overflows are detected and reported (Req #3), the grammar can be used for Yacc-implementations that generate for other languages (Req #2) and the generated code only depends on memset() and realloc().

The lexer generator: Ragel

Ragel is actually more than a lexer generator, it’s a “State Machine Compiler”. Having used it for the first time, I’m sure to return to it more often, it looks great.

Even if abused for “just” generating lexers, the generated code is great. A few (ROMable) tables, the logic code and the action code interweaved and that’s all. No allocations, no dependencies, nothing.

Ragel can also generate for different languages, fulfilling Req #2.

Having them work in tandem

There’s not much more to be said, check out the example I’ve put up at GitHub.

It shows how to use them together and how to have multiple instances of a parser.

That’s it! Like always, questions and comments welcome.

Other tools

I’ve looked at other tools which I eventually didn’t use. That’s what I found out about them:

Coco/R

Coco has a nice EBNF-syntax and supports many languages (Req #2).

It generates a recursive descent (LL-) parser, which is the kind of parser you would write by hand. Hence, Coco/R generates really nice and readable code with the action-code cleanly interweaved.

Unfortunately, the lexer is unusable (C++ variant, version 2012-01-02) for this case:

Allocates a lot of heap (60 kB)
Limited to wchar_t-Strings with its own string-functions (instead of std::basic_string<T>)
Lots of allocations
Dependency on wprintf, wcs*

Coco/R supports custom lexers, so I exchanged Coco’s lexer with a Ragel-lexer.

Eventually I found out that the generated parser may use a lot of stack, sometimes over 6 kB – yes, that can be too much.

The problem is recursive descent parsers in combination with a grammar that has lots of precedence-levels, like e.g. C-style expressions with 15 or more of them.

One precedence level results in one grammar rule which in turn results in one function call. If there are over 15 calls until you finally hit a terminal and then have a rule with parentheses which make the parser start at the top rule again and then there may be another parenthesis which… well, you get the idea.

This is where the 6 kB come from: The stack frame of one method (i.e. grammar rule) may be 64 bytes or more in size (this, parameters, local vars, registers) and the parser-input can easily result in a 100 frames deep callstack.

Another serious drawback is that there’s no built-in way of detecting stack overflows (Req #3). The stack may just overflow and that’s that.

Others

ANTLR: v3 supports C/C++, but it seems to have a big runtime
SableCC: Supports C/C++ with “Alternative output”, but not with ROMability in mind (tables aren’t const)
(f)lex: depends on stdio and FILE and all that stuff

CMakeDebHelper: Using debhelper with CMake/CPack

2013-11-30T00:00:00+00:00

CMake/CPack is great for quickly packaging your stuff for Debian.

However for instance, you may have programmed a daemon and want to let debhelper help you with update-rc.d-calls in your postinst/postrm-scripts. Also, you may want to easily package cron.d-rules. There’s a debhelper for everything.

CPack does not let you use them without further ado, as it’s built to not rely on anything else.

Here’s my way of making CPack and debhelper really good friends: CMakeDebHelper.

Arguably, it would be more like the “Debian-way” to actually use Debian’s tools for packaging. However, this basically assumes that you’re in the role of a package-maintainer, which may not always be the case.

In my case, I’ve written software that is deployed to clients as a Debian-package and I prefer my build-system (CMake) to be in control of building and packaging.

Since you’ve found this post, you probably already know about the benefits of debhelpers. For those who don’t, here’s an example:

When packaging for Debian, you usually type up scripts to execute after installation (“postinst”) or after removal (“postrm”), among others. A package containing a daemon usually includes stuff for managing the “rc.d”-links through calling “update-rc.d”.

No need to make up that code yourself, the debhelper “dh_installinit” can add that code for you! Look what it generates for example:

# Automatically added by dh_installinit
if [ -x "/etc/init.d/cmakedh" ]; then
        update-rc.d cmakedh defaults >/dev/null
        invoke-rc.d cmakedh start || exit $?
fi
# End automatically added section

It also generates the script to run before removal (“prerm”) itself:

#!/bin/sh
set -e
# Automatically added by dh_installinit
if [ -x "/etc/init.d/cmakedh" ]; then
        invoke-rc.d cmakedh stop || exit $?
fi
# End automatically added section

If you’re anything like me, you’ll really want to leverage functionality like that!

There’s much more stuff, have a look at the Debian New Maintainers’ Guide for more information.

So, with my CMakeDebHelper you’ll easily go from such an input:

.
├── CMakeLists.txt
├── CPackConfig.cmake
├── cmake
│   ├── CMakeDebHelper.cmake
│   └── CMakeDebHelperInstall.cmake
├── debian
│   ├── CMakeLists.txt
│   ├── cmakedh.cron.d.in
│   ├── cmakedh.init.in
│   ├── cmakedh.postinst.in
│   └── cmakedh.postrm.in
└── src
    ├── CMakeLists.txt
    └── main.c

to a Debian-package with these contents:

new debian package, version 2.0.
 size 3690 bytes: control archive=850 bytes.
      40 bytes,     2 lines      conffiles            
     201 bytes,    10 lines      control              
     154 bytes,     3 lines      md5sums              
     458 bytes,    30 lines   *  postinst             #!/bin/sh
     425 bytes,    29 lines   *  postrm               #!/bin/sh
     169 bytes,     7 lines   *  prerm                #!/bin/sh
 Package: cmakedh
 Version: 1.0.0
 Section: devel
 Priority: optional
 Architecture: i386
 Depends: libc6 (>= 2.0), make, rsync, adduser, liblog4cpp5, zlib1g, libboost-system1.49.0, libboost-regex1.49.0, libboost-filesystem1.49.0
 Installed-Size: 6
 Maintainer: seb <seb@asdf>
 Description: Package summary

drwxr-xr-x root/root         0 2013-11-30 16:01 ./usr/
drwxr-xr-x root/root         0 2013-11-30 16:01 ./usr/bin/
-rwxr-xr-x root/root      5360 2013-11-30 16:01 ./usr/bin/runme
drwxr-xr-x root/root         0 2013-11-30 16:01 ./etc/
drwxr-xr-x root/root         0 2013-11-30 16:01 ./etc/init.d/
-rwxr-xr-x root/root        35 2013-11-30 16:01 ./etc/init.d/cmakedh
drwxr-xr-x root/root         0 2013-11-30 16:01 ./etc/cron.d/
-rw-r--r-- root/root        24 2013-11-30 16:01 ./etc/cron.d/cmakedh

The ham and a working example can be found on GitHub, improvements welcome!

Running Logitech Media Server in a chroot on Debian Wheezy

2013-10-09T00:00:00+00:00

!!! This is outdated information. Left here for reference. !!!

Oh no, another chroot-tutorial! This time Logitech Media Server gets jailed. If you only have it running locally it’s not necessary to imprison it, but if you have it running on a public IP it is. Big software with plugins and open ports just creates this chroot-urge deep inside of me. It’s quick and easy, read on!

So, what do I want?

A minimal Debian at /var/local/squeezebox
A Logitech Media Server chrooted to that directory running as a certain user
Both the minimal Debian and the Media Server should be easily upgradeable through apt and dpkg

Installation

Install “cdebootstrap” and run these commands as root:

cdebootstrap -f minimal --include apt,apt-rdepends wheezy /var/local/squeezebox

Afterwards there’ll be a minimal Debian in /var/local/squeezebox.

Fetch the latest Media Server Debian package here and put it into our new chroot.

Then, start a chrooted bash by running this:

chroot /var/local/squeezebox /bin/bash -l

Then, we’ll do some setting up:

# To avoid some errors
export LANG=C.UTF-8
 
# Install the deb, there'll be unresolved dependencies!
dpkg -i logitechmediaserver_7.7.3_all.deb
 
# Make sure we're getting security updates
echo "deb http://security.debian.org wheezy/updates main" >> /etc/apt/sources.list
 
# Fetch available packages
apt-get update
 
# Make sure everything required by logitechmediaserver is pulled in
apt-get -f install

In the next step we’ll change the UID and GID of the user “squeezeboxserver” to the “real” UID:GID we want the server to be running as. In my case, it should be seb:seb (1000:1000 on my system).

Then (still in the chrooted shell):

# First, remember the original UID, it was 101 on my system
id -u squeezeboxserver

# The user is in group nogroup, let's put it into a dedicated group
addgroup squeezeboxserver
usermod -g squeezeboxserver squeezeboxserver

# Now change it to the ids it should be running as
usermod -u 1000 squeezeboxserver
groupmod -g 1000 squeezeboxserver

# Re-own its files to this uid:gid. Change 101 to whatever the command in line 2 spat out 
find / -uid 101 -exec chown squeezeboxserver:squeezeboxserver {} \;

As a precaution, let’s remove the suid-bit from all binaries that have it. The following command is meant to be run in the chrooted shell, be careful!

find / -perm +6000 -type f -exec chmod -s {} \;

Removing “unneeded” packages

This step is not recommended, it’s just an “academic exercise”. I just keep it here as reference. Go to the next step.

It removes all the packages that aren’t required for operation and this will break updating at some point. I was curious how to go about it and this is the result. Again, be careful, this is meant to be run in the chrooted shell!

# Get a list of packages required for the packages mentioned on the command line
apt-rdepends coreutils bash findutils sed apt dpkg libpam-runtime logitechmediaserver 2>/dev/null | grep -v '^ ' | sort | uniq > debs-needed
 
# Get a list of installed packages
dpkg-query -W -f='${Package}\n' | sort | uniq > debs-installed
 
# Remove those packages which are in debs-installed but not in debs-needed, possibly removing essential packages
dpkg -P --force-remove-essential `comm -23 debs-installed debs-needed`
 
apt-get clean

Starting the server

We’re going to start the server as the specific user directly, no root-privileges are ever needed. The following commands are run outside the chroot (hit Ctrl-D if you’re still in the chrooted shell).

There’s just one more thing to take care of, chowning the run-directory for the start-stop-daemon’s PID-file:

chown -R seb:seb /var/local/squeezebox/run/

Then we can start the server by running this as root:

chroot --userspec=1000:1000 --groups=1000 /var/local/squeezebox /etc/init.d/logitechmediaserver start

I prefer dropping the user-privileges at this stage. It’s probably safe without the userspec/groups-stuff because the start-script will drop its rights through start-stop-daemon anyway, but better safe than sorry.

Chroot’s man-page claims that names instead of numeric ids can be used, but it doesn’t seem to work. There’s one more catch: Without the –groups-parameter, the resulting proc will still have 0 in its groups! Bug or feature?

That should be all. If you want the server to be started at boot-time, a simple script like

#!/bin/sh
chroot --userspec=1000:1000 --groups=1000 /var/local/squeezebox /etc/init.d/logitechmediaserver $1

in /etc/init.d/ of the “real” filesystem should do the job.

For upgrading, simply launch a chrooted bash as shown above and use dpkg and apt-get as always.

That’s all for today!

Krawall GBA on GitHub

2013-07-18T00:00:00+00:00

Krawall GBA is an XM/S3M modplayer / soundsystem for the Gameboy Advance that I was developing 10 years ago. Back then, I successfully licensed it to quite some game developer studios and it was used in games like “The Sims”, “Spiderman”, “Lord of the Rings” and others.

I’ve removed the dust, updated it to be compatible with the latest devkitPro and rewritten its build-system, also I put it under the LGPL. Get it at GitHub.