1
0
Fork 0

initial commit

This commit is contained in:
Harald Hoyer 2020-05-19 18:20:18 +02:00
commit 429a176559
59 changed files with 4097 additions and 0 deletions

View file

@ -0,0 +1,5 @@
+++
transparent = true
sort_by = "date"
paginate_by = 5
+++

View file

@ -0,0 +1,41 @@
+++
title = "Single UEFI executable for kernel+initrd+cmdline"
date = 2015-02-25T15:55:16+00:00
+++
Lately Kay Sievers and David Herrmann created a UEFI
[loader stub](http://cgit.freedesktop.org/gummiboot/tree/src/efi/stub.c), which starts a linux kernel
with an initrd and a kernel command line, which are COFF sections of the executable.
This enables us to create single UEFI executable with a standard distribution kernel,
a custom initrd and our own kernel command line attached.
<!-- more -->
Of course booting a linux kernel
directly from the UEFI has been possible before with the kernel EFI stub. But to add an initrd and
kernel command line, this had to be specified at kernel compile time.
To demonstrate this feature
and have a useful product, I created a shell script, which creates a "rescue" image on Fedora with
the rescue kernel and rescue initrd. The kernel command line "rd.auto" instructs dracut to assemble
all devices, while waiting 20 seconds for device appearance "rd.retry=20" and drop to a final shell
because "root=/dev/failme" is specified (which does not exist of course). Now in this shell you can
fsck your devices, mount them and repair your system.
The shell script can be downloaded and viewed on
[github](https://github.com/haraldh/mkrescue-uefi).
To run the script, you will need to install gummiboot >= 46 and binutils.
```console
# yum install gummiboot binutils
```
Run the script:
```console
# bash mkrescue-uefi.sh BOOTX64.EFI
```
Copy `BOOTX64.EFI` to e.g. a USB stick to `EFI/BOOT/BOOTX64.EFI` in the FAT boot partition
and point your BIOS to boot from the USB stick. Voilà!
A rescue USB stick with just one file! :-)

View file

@ -0,0 +1,5 @@
+++
transparent = true
sort_by = "date"
paginate_by = 5
+++

View file

@ -0,0 +1,5 @@
+++
transparent = true
sort_by = "date"
paginate_by = 5
+++

View file

@ -0,0 +1,79 @@
+++
title = "libtool: getting rid of 180,000 sed forks"
date = 2015-03-05T08:10:36+00:00
+++
When compiling systemd on rawhide, we noticed a significant slowdown in compile time.
Investigating further, it turns out, that libtool forks an incredible amount of sed.
Running perf showed 30% of the "make all" was spent in bash. strace showed an execve of
sed with the same arguments 180,000 times!!!!!
<!-- more -->
Looking in the code reveals the culprit:
```bash
_G_unquoted_arg=`printf '%s\n' "$1" |$SED "$sed_quote_subst"` ;;
```
and
```bash
qECHO=`$ECHO "$ECHO" | $SED "$sed_quote_subst"`
```
`sed_quote_subst` looks like this:
```bash
sed_quote_subst='s|\([`"$\\]\)|\\\1|g'
```
This substitution can be easily done in bash, without forking sed. So I came up with a patch for libtool:
```diff
--- libtool.orig 2015-03-04 15:44:39.632703654 +0100
+++ libtool.fixed 2015-03-04 17:16:42.495983610 +0100
@@ -1607,7 +1607,15 @@
while test 0 -lt $#; do
case $1 in
*[\\\`\"\$]*)
- _G_unquoted_arg=`printf '%s\n' "$1" |$SED "$sed_quote_subst"` ;;
+ if test set = "${BASH_VERSION+set}"; then
+ _G_unquoted_arg=${1//\\/\\\\}
+ _G_unquoted_arg=${_G_unquoted_arg//\"/\\\"}
+ _G_unquoted_arg=${_G_unquoted_arg//\$/\\\$}
+ _G_unquoted_arg=${_G_unquoted_arg//\`/\\\`}
+ else
+ _G_unquoted_arg=`printf '%s\n' "$1" |$SED "$sed_quote_subst"`
+ fi
+ ;;
*)
_G_unquoted_arg=$1 ;;
esac
@@ -5756,7 +5764,11 @@
if test \"\$libtool_execute_magic\" != \"$magic\"; then
file=\"\$0\""
- qECHO=`$ECHO "$ECHO" | $SED "$sed_quote_subst"`
+ if test set = "${BASH_VERSION+set}"; then
+ qECHO=$(_G=$($ECHO "$ECHO");_G=${_G//\\/\\\\};_G=${_G//\"/\\\"};_G=${_G//\$/\\\$};_G=${_G//\`/\\\`};printf '%s\n' "$_G")
+ else
+ qECHO=`$ECHO "$ECHO" | $SED "$sed_quote_subst"`
+ fi
$ECHO "\
```
With that patch, I compared the runtime of a compile (with ccache on). Original libtool:
```console
$ time make -j4
[…]
real 5m23.880s
user 11m39.731s
sys 7m12.578s
```
Fixed libtool:
```console
$ time make -j4
[…]
real 2m47.325s
user 9m33.561s
sys 0m57.522s
```
The old version spent half of the time creating a pipe, forking sed and parsing the
output. Someone with connections to the libtool maintainers, please point them to
this problem! Reported as [bug 20006](http://debbugs.gnu.org/cgi/bugreport.cgi?bug=20006).

View file

@ -0,0 +1,5 @@
+++
transparent = true
sort_by = "date"
paginate_by = 5
+++

View file

@ -0,0 +1,5 @@
+++
transparent = true
sort_by = "date"
paginate_by = 5
+++

View file

@ -0,0 +1,357 @@
+++
title = "A Python Transaction Class"
date = 2015-10-13T10:20:43+00:00
+++
This is a repost of a blog post of 2008. Just for the reference :-)
This class allows sub-classes to commit changes to an instance to a history, and rollback to previous states.
<!-- more -->
The final class with an extension for `__setstate__` and `__getstate__` can be found here:
[transaction.py](/files/transaction.py)
and [transaction_test.py](/files/transaction_test.py).
Now to the story, that led to it: I had the need for a transaction class in python and browsing the python cookbook led me to a [small Transaction class](http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/551788).
```python
$ python
Python 2.5.1 (r251:54863, Oct 30 2007, 13:45:26)
[GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> class Transaction(object):
... def __init__(self):
... self.log = []
... def commit(self):
... self.log.append(self.__dict__.copy())
... def rollback(self):
... try:
... self.__dict__.update(self.log.pop(-1))
... except IndexError:
... pass
...
```
Ok, lets have some fun with it.
```python
>>> class A(Transaction):
... pass
...
>>> a = A()
>>> a.test = True
>>> a.commit()
>>> a.test = False
>>> a
'self.__dict__ = {'test': False, 'log': [{'test': True, 'log': [...]}]}'
>>> a.rollback()
>>> a
'self.__dict__ = {'test': True, 'log': []}'
```
Nice. Let's see if we can commit and rollback several times.
```python
>>> a = A()
>>> a.test = 1
>>> a.commit()
>>> a.test = 2
>>> a.commit()
>>> a.test = 3
>>> a
'self.__dict__ = {'test': 3, 'log': [{'test': 1, 'log': [...]}, {'test': 2, 'log': [...]}]}'
>>> a.rollback()
>>> a
'self.__dict__ = {'test': 2, 'log': [{'test': 1, 'log': [...]}]}'
>>> a.rollback()
>>> a
'self.__dict__ = {'test': 1, 'log': []}'
```
Ok.. works :) Let's try some lists.
```python
>>> a = A()
>>> a.test = [ 0, 1 ]
>>> a.commit()
>>> a.test.append(2)
>>> a
'self.__dict__ = {'test': [0, 1, 2], 'log': [{'test': [0, 1, 2], 'log': [...]}]}'
>>> a.rollback()
>>> a
'self.__dict__ = {'test': [0, 1, 2], 'log': []}'
```
Doh! Ok, someone mentioned that already in the comments. copy.deepcopy() is the key.
```python
>>> import copy
>>>
>>> class Transaction2(Transaction):
... def commit(self, **kwargs):
... self.log.append(copy.deepcopy(self.__dict__))
...
>>> class A(Transaction2):
... pass
...
>>> a = A()
>>> a.test = [ 0, 1 ]
>>> a.commit()
>>> a.test.append(2)
>>> a
'self.__dict__ = {'test': [0, 1, 2], 'log': [{'test': [0, 1], 'log': []}]}'
>>> a.rollback()
>>> a
'self.__dict__ = {'test': [0, 1], 'log': []}'
```
Ah, works. Very good. Now another check:
```python
>>> a = A()
>>> a.test = 1
>>> a.commit()
>>> a
'self.__dict__ = {'test': 1, 'log': [{'test': 1, 'log': []}]}'
>>> a.other = 2
>>> a
'self.__dict__ = {'test': 1, 'other': 2, 'log': [{'test': 1, 'log': []}]}'
>>> a.rollback()
>>> a
'self.__dict__ = {'test': 1, 'other': 2, 'log': []}'
>>> a.other
2
```
Oh, a leftover... seems like `self.__dict__` has to be cleared, before the update.
```python
>>> class Transaction3(Transaction2):
... def rollback(self, **kwargs):
... try:
... state = self.log.pop(-1)
... self.__dict__.clear()
... self.__dict__.update(state)
... except IndexError:
... pass
...
>>> class A(Transaction3):
... pass
...
>>> a = A()
>>> a.test = 1
>>> a.commit()
>>> a
'self.__dict__ = {'test': 1, 'log': [{'test': 1, 'log': []}]}'
>>> a.other = 2
>>> a
'self.__dict__ = {'test': 1, 'other': 2, 'log': [{'test': 1, 'log': []}]}'
>>> a.rollback()
>>> a
'self.__dict__ = {'test': 1, 'log': []}'
>>> a.other
Traceback (most recent call last):
File "", line 1, in
AttributeError: 'A' object has no attribute 'other'
>>>
```
Ah, works. Very good. Ok, more tests...
```python
>>> b = a.ncls
>>> a.ncls.test = True
>>> a.commit()
>>> a.ncls.test = False
>>> a.rollback()
>>> a
'self.__dict__ = {'ncls': 'self.__dict__ = {'test': True, 'log': []}', 'log': []}'
>>> b
'self.__dict__ = {'test': False, 'log': []}'
```
Oh, what if we work with "b", which stills holds the old value? Maybe we should commit() all our attributes also, traversing through them all? Ok, here is such a beast:
```python
>>> class TransactionNew1(object):
... def _docommit(self):
... if "log" not in self.__dict__:
... self.__dict__["log"] = list()
...
... self.__dict__["log"].append(copy.deepcopy(self.__dict__))
...
... def _dorollback(self):
... if "log" not in self.__dict__:
... return
... try:
... state = self.__dict__["log"].pop(-1)
... self.__dict__.clear()
... self.__dict__.update(state)
... except IndexError:
... pass
...
... def commit(self, **kwargs):
... # commit ourselves, then our childs
... self._docommit()
... if kwargs.get("deep", True):
... for child in self.__dict__.values():
... if isinstance(child, self.__class__):
... child.commit()
...
... def rollback(self, **kwargs):
... # rollback our childs, then ourselves
... if kwargs.get("deep", True):
... for child in self.__dict__.values():
... if isinstance(child, self.__class__):
... child.rollback()
... self._dorollback()
...
... def __repr__(self):
... return "'self.__dict__ = %s'" % self.__dict__
...
>>> class A(TransactionNew1):
... pass
...
>>> a = A()
>>> a.ncls = A()
>>> b = a.ncls
>>> a.ncls.test = True
>>> a.commit()
>>> a.ncls.test = False
>>> a.rollback()
>>> a
'self.__dict__ = {'ncls': 'self.__dict__ = {'test': True}', 'log': []}'
>>> b
'self.__dict__ = {'test': True, 'log': []}'
```
Ok... looks good, but we lost the reference. id(b) != id(a.ncls) ... ( update: this is fixed in the final version ) Working with it revealed also:
```python
>>> a = A()
>>> b = a
>>> for i in xrange(3):
... b.n = A()
... b.t = "test"
... b = b.n
... a.commit()
...
>>> a
'self.__dict__ = {'log': [{'n': 'self.__dict__ = {}', 'log': [], 't': 'test'}, {'n': 'self.__dict__ = {'log': [{'log': []}], 't': 'test', 'n': 'self.__dict__ = {}'}', 'log': [{'t': 'test', 'log': [], 'n': 'self.__dict__ = {}'}], 't': 'test'}, {'n': 'self.__dict__ = {'log': [{'log': []}, {'log': [{'log': []}], 't': 'test', 'n': 'self.__dict__ = {}'}], 't': 'test', 'n': 'self.__dict__ = {'log': [{'log': []}], 't': 'test', 'n': 'self.__dict__ = {}'}'}', 'log': [{'t': 'test', 'log': [], 'n': 'self.__dict__ = {}'}, {'t': 'test', 'log': [{'n': 'self.__dict__ = {}', 't': 'test', 'log': []}], 'n': 'self.__dict__ = {'t': 'test', 'log': [{'log': []}], 'n': 'self.__dict__ = {}'}'}], 't': 'test'}], 't': 'test', 'n': 'self.__dict__ = {'t': 'test', 'log': [{'log': []}, {'n': 'self.__dict__ = {}', 't': 'test', 'log': [{'log': []}]}, {'n': 'self.__dict__ = {'log': [{'log': []}], 't': 'test', 'n': 'self.__dict__ = {}'}', 't': 'test', 'log': [{'log': []}, {'log': [{'log': []}], 't': 'test', 'n': 'self.__dict__ = {}'}]}], 'n': 'self.__dict__ = {'t': 'test', 'log': [{'log': []}, {'n': 'self.__dict__ = {}', 't': 'test', 'log': [{'log': []}]}], 'n': 'self.__dict__ = {'log': [{'log': []}]}'}'}'}'
>>> len(str(a))
1192
```
Hmm... seems strange.. Ah, self.log was also copied with copy.deepcopy(). So, we have multiple useless copies. Let's "pop()" the state from self.__dict__ before the deepcopy.
```python
>>> class TransactionNew2(TransactionNew1):
... def _docommit(self):
... if "log" in self.__dict__:
... oldstate = self.__dict__.pop("log")
... else:
... oldstate = None
... state = copy.deepcopy(self.__dict__)
... if oldstate:
... state["log"] = oldstate
... self.__dict__["log"] = state
... def _dorollback(self):
... if "log" not in self.__dict__:
... return
... try:
... state = self.__dict__["log"]
... self.__dict__.clear()
... self.__dict__.update(state)
... except IndexError:
... pass
...
>>> class A(TransactionNew2):
... pass
...
>>>
>>> a = A()
>>> b = a
>>> for i in xrange(3):
... b.n = A()
... b.t = "test"
... b = b.n
... a.commit()
...
>>> a
'self.__dict__ = {'log': {'log': {'log': {'t': 'test', 'n': 'self.__dict__ = {}'}, 't': 'test', 'n': 'self.__dict__ = {'log': {}, 't': 'test', 'n': 'self.__dict__ = {}'}'}, 't': 'test', 'n': 'self.__dict__ = {'log': {'t': 'test', 'n': 'self.__dict__ = {}'}, 't': 'test', 'n': 'self.__dict__ = {'log': {}, 't': 'test', 'n': 'self.__dict__ = {}'}'}'}, 't': 'test', 'n': 'self.__dict__ = {'t': 'test', 'log': {'log': {'t': 'test', 'n': 'self.__dict__ = {}'}, 't': 'test', 'n': 'self.__dict__ = {'log': {}, 't': 'test', 'n': 'self.__dict__ = {}'}'}, 'n': 'self.__dict__ = {'t': 'test', 'log': {'t': 'test', 'n': 'self.__dict__ = {}'}, 'n': 'self.__dict__ = {'log': {}}'}'}'}'
>>> len(str(a))
671
```
Ok, saved us a bit of state length. The final version has:
```python
>>> a
'self.__dict__ = {'__l': {'__l': {'__l': {'t': 'test'}, 't': 'test'}, 't': 'test'}, 't': 'test', 'n': 'self.__dict__ = {'__l': {'__l': {'t': 'test'}, 't': 'test'}, 't': 'test', 'n': 'self.__dict__ = {'__l': {'t': 'test'}, 't': 'test', 'n': 'self.__dict__ = {'__l': {}}'}'}'}'
>>> len(str(a))
275
```
Now another thing:
```python
>>> a = A()
>>> a.n = a
>>> a.commit()
File "/usr/lib64/python2.5/copy.py", line 162, in deepcopy
y = copier(x, memo)
RuntimeError: maximum recursion depth exceeded
```
... Oh, oh! Recursion in commit()... Now we have to check, if we have been there.
```python
>>> class TransactionNew3(TransactionNew2):
... def _checksetseen(self, seen):
... if id(self) in seen:
... import sys
... sys.stderr.write("Recursion detected... \n")
... return True
... seen.add(id(self))
... return False
...
... def commit(self, **kwargs): # pylint: disable-msg=W0613
... seen = kwargs.get("_commit_seen", set())
... if self._checksetseen(seen):
... return
... # commit ourselves, then our childs
... self._docommit()
... if kwargs.get("deep", True):
... for child in self.__dict__.values():
... if isinstance(child, self.__class__):
... child.commit(_commit_seen = seen)
...
... def rollback(self, **kwargs):
... seen = kwargs.get("_rollback_seen", set())
... if self._checksetseen(seen):
... return
... # rollback our childs, then ourselves
... if kwargs.get("deep", True):
... for child in self.__dict__.values():
... if isinstance(child, self.__class__):
... child.rollback(_rollback_seen = seen)
... self._dorollback()
...
>>>
>>> class A(TransactionNew3):
... pass
...
>>> a = A()
>>> a.n = a
>>> a.commit()
Recursion detected...
>>>
```
The final class with an extension for `__setstate__` and `__getstate__` can be found here: [transaction.py](https://harald.hoyer.xyz/files/transaction.py) and [transaction_test.py](/files/transaction_test.py). Have fun with it :-)

View file

@ -0,0 +1,5 @@
+++
transparent = true
sort_by = "date"
paginate_by = 5
+++

5
content/2015/_index.md Normal file
View file

@ -0,0 +1,5 @@
+++
transparent = true
sort_by = "date"
paginate_by = 5
+++