Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts

Feb 7, 2017

Jenkins as Code

I saw a couple of talks last week, and learned about several ways of automating Jenkins CI.

The problem being solved is: if you automate your builds and tests, why still click the Jenkins web UI by hand? Script it instead.

Jenkins Job DSL, which is based on Groovy (a JVM language). Another topic was Jenkins Pipeline which helps managing many jobs that depend on each other.

In Test Driven Infrastructure, Yury Tsarev presents, among many other things, Jenkins Job Builder. JJB takes descriptions written in YAML or JSON and translates them to Jenkins API with Python.

Jan 30, 2017

Capturing and Decoding Lego Mindstorms EV3 Bluetooth Communication

The Lego Mindstorms EV3 robots can be controlled with an Android app (Lego Mindstorms Commander) communicating with the brick via Bluetooth. The command protocol is documented by Lego in the EV3 Communication Developer Kit and the commands themselves in the EV3 Firmware Developer Kit (get them from Mindstorms Downloads).

I wondered what exactly goes on and I decided to capture the communication and decode it, to learn both about Bluetooth and about the details of the EV3 protocol.

Good Robot. Good Robot!!

I succeeded and made a couple of useful tools along the way:

See also the previous post about sending data (EV3 commands) over USB.

Outline

  1. Enable Android Bluetooth logging
  2. Run the Commander app and exercise the robot a bit
  3. Transfer the log to a PC
  4. Extract the serial data (RFCOMM) from the Bluetooth dump
  5. Decode the EV3 protocol
  6. Disassemble the EV3 instructions

1. Enable Android Bluetooth Logging

  • Open Settings
  • In the System section, choose Developers (this needs to be enabled first by tapping Build number 7 times)
  • Enable Bluetooth HCI Log

2. Run the Commander app and exercise the robot a bit

3. Transfer the log to a PC

On the phone/tablet:

  • Open Settings
  • System > Developers
  • Disable Bluetooth HCI Log

Connect to the PC with a USB cable.

My older Android phone offered to mount its storage as a USB disk drive, but the newer one no longer has that option, offering MTP instead. I transfered the log file with a KDE tool:

$ kioclient cp 'mtp:/Xperia Z3/Interní úložiště/btsnoop_hci.log' .

4. Extract the serial data (RFCOMM) from the Bluetooth dump

The tool I made for this is btsnoop-decode.rb.

I learned the bare minimum needed about Bluetooth so it is very likely the tool only works for this specific use case.

Originally I opened the btsnoop log with Wireshark and guessed my way through the BT protocol layers. In the end the RFCOMM length field was harder than usual to guess and half of my packets were wrong. So I resorted to finding the appropriate part of the Linux kernel source to find out the format.

5+6. Decode the EV3 protocol and dissassemble the EV3 instructions

The people of the ev3dev project have already produced a disassembler which we will use in the next step. But that one assumes you start with a program file (RBF).

Here we have a log containing not only the usual RBF instructions but also System Commands.

I made an ugly hack of the lmsdisasm tool and arrived at a version that disassembles the log produced by the RFCOMM extractor.

Play time

The above experiments enabled me to put together a little script that can control the robot from a Linux terminal, having it ride around and even speak a custom sound file: lethargic-ministers/lms.py.

Jan 2, 2017

USB Communication with Python and PyUSB

Say we have a robot with a USB connection and command documentation. The only thing missing is knowing how to send a command over USB. Let's learn the basic concepts needed for that.

General Bunny catching Pokemon

Installing the Library

We'll use the pyusb Python library. On openSUSE we install it from the main RPM repository:

sudo zypper install python-usb

On other systems we can use the pip tool:

pip install --user pyusb

Navigating USB Concepts

To send a command, we need an Endpoint. To get to the endpoint we need to descend down the hierarchy of

  1. Device
  2. Configuration
  3. Interface
  4. Alternate setting
  5. Endpoint

First we import the library.

#!/usr/bin/env python2

import usb.core

The device is identified with a vendor:product pair included in lsusb output.

Bus 002 Device 043: ID 0694:0005 Lego Group

VENDOR_LEGO = 0x0694
PRODUCT_EV3 = 5
device = usb.core.find(idVendor=VENDOR_LEGO, idProduct=PRODUCT_EV3)

A Device may have multiple Configurations, and only one can be active at a time. Most devices have only one. Supporting multiple Configurations is reportedly useful for offering more/less features when more/less power is available. EV3 has only one configuration.

configuration = device.get_active_configuration()

A physical Device may have multiple Interfaces active at a time. A typical example is a scanner-printer combo. An Interface may have multiple Alternate Settings. They are kind of like Configurations, but easier to switch. I don't quite understand this, but they say that if you need Isochronous Endpoints (read: audio or video), you must go to a non-primary Alternate Setting. Anyway, EV3 has only one Interface with one Setting.

INTERFACE_EV3 = 0
SETTING_EV3 = 0
interface = configuration[(INTERFACE_EV3, SETTING_EV3)]

An Interface will typically have multiple Endpoints. The Endpoint 0 is reserved for control functions by the USB standard so we need to use Endpoint 1 here.

The standard distinguishes between input and output endpoints, as well as four transfer types, differing in latency and reliability. The nice thing is that the Python library nicely allows to abstract all that away (unlike cough Ruby cough) and we simply say to write to a non-control Endpoint.

ENDPOINT_EV3 = 1
endpoint = interface[ENDPOINT_EV3]

# make the robot beep
command = '\x0F\x00\x01\x00\x80\x00\x00\x94\x01\x81\x02\x82\xE8\x03\x82\xE8\x03'
endpoint.write(command)

Other than Robots?

Robots are great fun but unfortunately they do not come bundled with every computer. Do you know of a device that we could use for demonstration purposes? Everyone has a USB keyboard and mouse but I guess the OS will claim them for input and not let you play.

What Next

The Full Script

Jul 21, 2009

HackWeek: NetworkManager Python library

It is Hack Week again, a time when we venture into unexplored territories. So why am I returning to cnetworkmanager?

It is because I am doing a complete rewrite, based on an easy-to-use library that hides some rough edges in dbus-python. If you ever thought NM needed some programmatic prodding but were put off by the tedious details, the library might be just the thing for you. For me, it is also an opportunity to learn more Python.

Compare how you get the interface names.
Before:
#! /usr/bin/python
import dbus
bus = dbus.SystemBus()
NM = 'org.freedesktop.NetworkManager'
nm = bus.get_object(NM, '/org/freedesktop/NetworkManager')
for dev_opath in nm.GetDevices(dbus_interface = NM):
dev = bus.get_object(NM, dev_opath)
print dev.Get(NM + '.Device', 'Interface',
dbus_interface='org.freedesktop.DBus.Properties')
After:
#! /usr/bin/python
from networkmanager.networkmanager import NetworkManager
nm = NetworkManager()
for d in nm.GetDevices():
print d["Interface"]
It is in the librarize branch of the git repo. Beware, the code is still messy. Getting it:
git clone git://repo.or.cz/cnetworkmanager.git cnetworkmanager.librarize
cd cnetworkmanager.librarize
git checkout -b librarize origin/librarize
./cnetworkmanager -h

Jun 2, 2009

A Minimal Build Service OSC Plugin

I am trying to make OBS cooperate with my old release scripts*. So far I have this minimal plugin for osc to share:
$ mkdir ~/.osc-plugins
$ cd $_
$ cat > releasecheck.py <<EOF
def do_releasecheck(self, subcmd):
print "Release early, release often."
EOF
$ osc releasecheck
Release early, release often.
*) They are not released, bad me; but they rely on the internal directory layout anyway.

Jul 14, 2008

cnetworkmanager 0.7.1

Cnetworkmanager is a command-line client for NetworkManager, intended to supplement and replace the GUI applets. So far it is a single python script. What is new in version 0.7.1:
  • it does not need a configuration file anymore:
    cnetworkmanager -C publicnet
    cnetworkmanager -C myessid --wep-hex 112234445566778899aabbccdd
    cnetworkmanager -C another --wpa-psk-hex \
     112233445566778899aabbccddeeff00112233445566778899aabbccddeeff00
  • it works with NM 0.6 (tested on Ubuntu 8.04 Hardy and a pre-release OLPC) in addition to the older support for NM 0.7pre (tested on openSUSE 11.0)
What is still left to do:
  • sooner:
    • specifying the key as a non-hex passphrase
    • reading the configuration stored by the GNOME nm-applet
    • possibility to quit after a connection is established
  • later:
    • more encryption schemes (WPA2?)
    • more connection types (dial-up, VPN)

Jun 8, 2008

KNotify Client

Korshak asked how to do emerge -uND --world && knotify "Done!". I thougt it would be a piece of cake, but apparently not.
The short answer, zypper in libnotify; notify-send Done, has a disadvantage of requiring a fair amount of GNOME software. But it does use a proposed freedesktop.org standard.
Then I find the method org.kde.KNotify.event, but its nice signature (ssavsayasx) means dbus-send (dbus-1-1.2.1-12) cannot invoke it (it cannot pass empty arrays, and it cannot wrap variants in arrays). No problem, let's use Python. But then we are blocked by knotify4 (kdebase4-runtime-4.0.4-19) reporting a wrong signature via the introspection interface.
# killall knotify4; cp /usr/bin/knotify4{,.bak}
# sed 's/type="a(ss)"/ type = "av"/g' /usr/bin/knotify4.bak > /usr/bin/knotify4
Now which event should we use so that the user actually sees the message? There are many of them, with user configurable actions. warning/kde seems to work out of the box.
#! /usr/bin/python
import sys
import dbus
m = sys.argv[1]
kn = dbus.SessionBus().get_object("org.kde.knotify", "/Notify")
i = kn.event("warning", "kde", [], m, [0,0,0,0], [], 0,
    dbus_interface="org.kde.KNotify")
That did it. But I still hope that there is a simple solution for KDE4 that I have missed.

May 30, 2008

try cnetworkmanager 0.3

cnetworkmanager is a command line interface for Network Manager. It is still lacking many features but I decided to announce it now, when it can:

  1. connect to WiFi networks
  2. read the configuration file of KNetworkManager

It is working for me on RC1 of openSUSE-11.0, which has NetworkManager-0.7.0.r3685-3. It is my first project in Python, so the code is somewhat cumbersome. Thanks to Mišo for getting me started (idea.o.o, abclinuxu.cz), and Bille for the API docs.

Regarding related projects, I have seen network-manager-cli, but it is for 0.6 only and the developers were only active last August, so I decided that I should first write some working code for 0.7.

I am interested in comments, bugs, feature requests, patches, any feedback.

Here are some screenshots ;-)

$ cnetworkmanager -h
cnetworkmanager 0.3 - Command Line Interface for NetworkManager
Options:
-d, --dev list devices
-c, --actcon list active connections
-s, --syscon list system connection settings
-u, --usrcon list user connection settings (can CRASH nm-applet)
-a, --ap list found access points
-n, --nets list found wireless networks
-C<net>, --connect=<net> connect to a wireless network (using knetworkmanagerrc)
-m, --monitor loop to show dbus signals

Listing the networks:

$ cnetworkmanager -n
cnetworkmanager 0.3 - Command Line Interface for NetworkManager
Wifi Networks:
49: onenet (54Mb, protected)
46: IT (54Mb, protected)
48: TiborBrunclik (11Mb, protected)
46: linksys (54Mb)
46: IT (54Mb, protected)
46: 23 (54Mb)

May 5, 2008

cnetworkmanager 0.1

Thanks to Mišo for getting me started on this...
$ cnetworkmanager -d -c
cnetworkmanager 0.1 - Command Line Interface for NetworkManager
state: CONNECTED
wifi: 1
wifi hw: 1
/org/freedesktop/Hal/devices/net_00_02_3f_0d_7d_93
Udi: /org/freedesktop/Hal/devices/net_00_02_3f_0d_7d_93
Interface: eth0
Driver: 8139too
Ip4Address: 268702730
Capabilities: NM_SUPPORTED,CARRIER_DETECT
Dev State: FAILED
Dev Type: 802_3_ETHERNET
HwAddress: 00:02:3F:0D:7D:93
Speed: 100
Carrier: 1
/org/freedesktop/Hal/devices/net_00_11_d8_70_00_2f
Udi: /org/freedesktop/Hal/devices/net_00_11_d8_70_00_2f
Interface: wlan0
Driver: rt2500pci
Ip4Address: 1476989962
Capabilities: NM_SUPPORTED
Dev State: FAILED
Dev Type: 802_11_WIRELESS
Dev Mode: INFRA
Wifi Capabilities: RSN,CIPHER_WEP40,CIPHER_WEP104,CIPHER_TKIP,CIPHER_CCMP,WPA
HwAddress: 00:11:D8:70:00:2F
Bitrate: 1000
ActiveAccessPoint: /org/freedesktop/NetworkManager/AccessPoint/4
Active Connections
/org/freedesktop/NetworkManager/ActiveConnection/8
ServiceName: org.freedesktop.NetworkManagerSystemSettings
Connection: /org/freedesktop/NetworkManagerSettings/0
SpecificObject: /
SharedServiceName:
SharedConnection: /
Devices: dbus.Array([dbus.ObjectPath('/org/freedesktop/Hal/devices/net_00_02_3f_0d_7d_93')], signature=dbus.Signature('o'), variant_level=1)
/org/freedesktop/NetworkManager/ActiveConnection/9
ServiceName: org.freedesktop.NetworkManagerUserSettings
Connection: /org/freedesktop/NetworkManagerSettings/Connection/0
SpecificObject: /org/freedesktop/NetworkManager/AccessPoint/4
SharedServiceName:
SharedConnection: /
Devices: dbus.Array([dbus.ObjectPath('/org/freedesktop/Hal/devices/net_00_11_d8_70_00_2f')], signature=dbus.Signature('o'), variant_level=1)