Fiddling with Nexus 4 boot image

TLDR; How to modify any system to set ro.debuggable=1 without rebuilding it from source. This setting will make any apk debuggable on the device.

Get the existing boot image off the phone

dd if=/dev/block/mmcblk0p6 of=/mnt/sdcard/boot.img # on the phone
adb pull /mnt/sdcard/boot.img # on your computer

/dev/block/mmcblk0p6 is Nexus 4's boot partition.

Install abootimg from https://github.com/coruus/abootimg. The rest of the process below is stolen from this page.

Extract and unpack initrd

mkdir boot 
cd boot
abootimg -x /tmp/boot.img

mkdir initrd
cd initrd
cat ../initrd.img | gunzip | cpio -vid

Edit default.prop, setting anything you want, including ro.debuggable=1.

Repack initrd and boot image

cd initrd
find . | cpio --create --format='newc' | gzip > ../myinitrd.img
 
cd ..
abootimg --create myboot.img -f bootimg.cfg -k zImage -r myinitrd.img

Flash to phone

adb reboot-bootloader
fastboot flash boot myboot.img

Big data analytics with grep and sed

This was written a couple of years ago, when a gigabyte of logs was reasonably "large scale". Nevertheless, the approach still works for many log monitoring tasks.

Part 1.

A few jobs ago I was on a team tasked with creating a system to monitor security events for a Swiss bank with 20k+ (all physical!) Unix hosts plus 60k Windows desktops and servers.

Hadoop did not exist yet outside of Yahoo research labs, so we had to do with MySQL, a bit of C and Perl. By the way, the resulting system kept chugging along for at least 5 years, if not 10, before finally succumbing to the likes of ArcSight.

We already had a centralised syslog setup, which produced tons of logs. To start with, we needed to come up with:

• a list of things to ignore, 
• a list of things to definitely alert on, 
• and an anomaly detection algorithm. 

This turned out simpler than it sounded (hurray for 20-20 hindsight!), thanks to some ideas borrowed from Marcus Ranum's "Artificial Ignorance" and "Never Before Seen"

Part 2.

Back at my current job, I'd spent a couple of hours digging into auth.log from our (then) new and shiny SaaS infrastructure manager node (~ 1 Gb), and it summed up nicely into 38k of "interesting" events. Once you've created your own stoplist, you can run the filtered output in a terminal in background. It contains 10-20 messages an hour:

• Manual, as opposed to scripted jobs, ssh logins,
• Unscheduled scripts,
• Adhoc fix work,
• Auth failures,
• Various unique ("never before seen") alarms.

I'm not including a sample of the filtered results - try this at home instead.

What I did was a dozen of iterations of

cat auth.* | grep -v -f stoplist.txt |sed -f sedcommands.txt |  sort | uniq -c | sort -r -n > stuff

starting with an empty stoplist and the following file sedcommands.txt:

s/^.\{15\}\ [a-z0-9.-]*\ //
s/\[[0-9]*\]://

This cuts off the initial part of the log line and summarises similar lines.  Looking at the sorted results, I add a few lines to the stoplist to get rid of the most uninteresting from security/anomaly point of view entries (usually top hits), rinse, repeat.

The final stoplist (slightly censored):

last message repeated .* times
Accepted publickey for .* from 59\.167\.XX\.XX
Accepted publickey for .* from 10\.65\..*
Accepted publickey for .* from 207\.223\.XX\.XX
session opened for user root by (uid=0)
session opened for user postgres by (uid=0)
session opened for user YYYYYYYY by (uid=0)
session opened for user www-data by (uid=0)
session closed for user
Received disconnect from .*: 11: disconnected by user
Accepted publickey for YYYYYYYY from 67\.221\.XX\.XX
USER=root ; COMMAND=/usr/lib/nagios/plugins/uc_.*
USER=root ; COMMAND=/usr/share/pyshared/pycm/bin/nagios/
USER=root ; COMMAND=/usr/lib/container-manager/.*
USER=root ; COMMAND=/opt/.*
USER=root ; COMMAND=/usr/bin/ZZZZZZ
PWD=/usr/share/pyshared/pycm/bin/vz ; USER=root ;
PWD=/usr/share/pyshared/pycm/bin/vz ; USER=root ;
PWD=/opt/unicorn-repl-tools/var/queue ; USER=root ; COMMAND=/bin/rm \./.*\.net
PWD=/opt/unicorn-repl-tools/var/queue ; USER=root ; COMMAND=/bin/rm \./.*\.com
PWD=/home/hal-uc-1 ; USER=root ; COMMAND=/usr/share/pyshared/pycm/
running '/sbin/halt' with root privileges on behalf of 'root'



This is only for illustration - do not copy the result, the point of this exercise is for you to come up with your own that is best fit for your environment. If you deal with extremely complex logfiles, you could throw in some AWK :)

In all, it took about 2 hours in total from getting the logs to producing the summary, including the remembering how this stuff was done.

Fixing VMWare pain in the ass performance

A tech-ish post for a change. Just for the Googles of the world.

If you are running more than one VM on the same host, and performance is hell, you need to:

$ mkdir ~/.vmware
$ echo "sched.mem.pshare.enable = FALSE" >>~/.vmware/config

and launch VMWare. This makes VMWare stop constantly comparing memory blocks in hopes of finding common ones so that it can fake more memory on the page level. It will start using real RAM. Massive improvement, in my experience. You will obviously need more RAM.

Other good settings are:

MemTrimRate = 0
prefvmx.useRecommendedLockedMemSize = "TRUE"
prefvmx.minVmMemPct = "100"

How to set up an ARM VM for Open Security Training's "Intro to ARM" course

Don't you hate it when you can't follow the video and end up just staring at the slides?

Folks at http://opensecuritytraining.info/IntroARM.html published a great course about ARM assembly. Chances are, if you are watching the course, you'll want to follow along with the class' labs and examples. The speaker uses a QEMU VM, which, alas, is not provided with the course. In fact, he uses a QEMU VM inside an Ubuntu VM inside, umm, Windows something? (If you read Russian, check this out :)

Not to worry, thanks to Linaro, it's quite easy to set one up without getting involved into a complex cross-compiling project.

Here's a compressed version how to do it. This works on Macs and Linux-es (or is it Linuces?). None of this is original, I took the info off a bunch of web pages, links below.

ARM VM setup

1. Install QEMU.

sudo apt-get install qemu qemu-system (on Linux)
brew install qemu (on Mac)

You may need to install other packages, depending on what OS you're on, different Linux flavours package QEMU differently. Check that you have qemu-system-arm command on your system, if not:

apt-cache search qemu-system-arm

to see what other packages you're missing. Another utility "qemu-img" will come in handy later, check that it is present as well. Check that your QEMU is at least v1.0 - Debian, for example, has very old builds by default.

2. Now you're ready to get a Linaro image running. Here are slightly modified instructions from https://wiki.linaro.org/PeterMaydell/QemuVersatileExpress. Versatile Express is a good choice of a modern ARM system to emulate since form Mac's and Linux' QEMU can do it.

Get image:
wget http://releases.linaro.org/images/12.03/oneiric/nano/vexpress-a9-nano.img.gz
gunzip vexpress-a9-nano.img.gz

Extract kernel and initrd:

(IMG=vexpress.img ; if [ -e "$IMG" ] ; then sudo mount -o loop,offset="$(file "$IMG" | awk 'BEGIN { RS=";"; } /partition 2/ { print $7*512; }')" -t auto "$IMG" /mnt/mnt; else echo "$IMG not found"; fi )

sudo cp -vr /mnt/mnt/boot .
sudo chown -R youruser:youruser boot
sudo umount /mnt/mnt

You end up with a number of images in .boot:

abi-3.3.0-1800-linaro-lt-vexpress-a9         
config-3.3.0-1800-linaro-lt-vexpress-a9      
initrd.img-3.3.0-1800-linaro-lt-vexpress-a9
System.map-3.3.0-1800-linaro-lt-vexpress-a9
vmlinuz-3.3.0-1800-linaro-lt-vexpress-a9

Copy the original vexpress image, vmlinuz.* and initrd.* somewhere where you want to keep your VM.

Run image:

qemu-system-arm -kernel vmlinuz-3.3.0-1800-linaro-lt-vexpress-a9 -M vexpress-a9 -cpu cortex-a9 -serial stdio -m 1024 -initrd initrd.img-3.3.0-1800-linaro-lt-vexpress-a9 -append 'root=/dev/mmcblk0p2 rw mem=1024M raid=noautodetect console=tty1 rootwait vmalloc=256MB devtmpfs.mount=0' -redir tcp:5022::22 -sd vexpress.img

This should bring up a black window with a Linux penguin in the top left corner and a usual boot sequence.

3. The image is configured with root autologin, there is a "linaro" user with "linaro" password as well. If you wish to autologins kill "/etc/init/openvt.conf" and edit "/etc/default/autogetty" to disable any autologin.

The qemu command above also maps the VM's port 22 to localhost:5022 so that you can use scp to copy files after sshd is installed like this:

scp -P 5522 projects.tar.gz linaro@localhost:~

4. Install tools inside the image:

apt-get install aptitude
aptitude install binutils gcc gdb build-essential vim openssh-server

...and whatever else looks missing. Beware that the linaro image comes with little free disk space.

apt-get clean

might save you a few megabytes, alternatively see next point.

5. Optional - upsize the image (make a copy first just in case):

qemu-img resize vexpress.img +512M

Then use parted or gparted (not available on Mac, unfortunately) on the image to actually resize the file system. It looks like gparted gets confused (those GUI tools!) - check out http://www.ros.org/wiki/groovy/Installation/Linaro/Source for how to deal with it.

Mac users might want to copy the .img file to a linux box/VM and run parted there. There is probably a contrived way using fdisk and resize2fs

6. Even more optional and probably does not work:

apt-get install cpufrequtils

- might speed the VM up a litte. Or it might not, I am not sure this util even works in a VM environment.

Tips and errata for the course

1. Lab 3 Interrupts is not in the projects.tar.gz :( Day 2 Part 2 video talks about it at length, though.
2. There are answers inside ".solutions" directory
3. Day 2 Part 1 around 41:30 (slide 159) is incorrect (I think) about the meaning of the caret in
   LDMFD sp!, {r0-r12,lr}
   - it means that the user mode registers are loaded and not the banked registers

Vim for code reading

GUI-based IDEs are nice tools for reading code, when properly set up. SourceInsight is probably the best based on the combination of efficiency/quality/price.

What if you find yourself with only a (colour) text console? I've put together a small .vimrc to make reading code a nicer experience in those situations.

If you are new to Vim, check out the README in https://github.com/agelastic/vim-reading for basic commands. Those are all you need to read and navigate any code in Vim.

.vimrc in that repo does the following:

• Ensures we are running a vim (and not vi) in a colour console
• Turns on search highlighting and 'search as you type'
• Sets detailed status lines
• Turns on code folding and makes initial state 'unfold all', which I like more than the default 'fold all on opening file'
• Turns saving some info on exiting file
• Lets you use Tab for vim command completion
• Maps Shift-Up and Shift-Down to move between Vim windows
• Maps Ctrl-X to "exit vim"

There are several plugins that look useful for this task, but I'm not adding them to keep the config lightweight.

If you want do explore those plugins, here's a bunch of links:

http://www.vim.org/scripts/script.php?script_id=610

http://www.vim.org/scripts/script.php?script_id=1581

http://www.vim.org/scripts/script.php?script_id=1213

http://www.vim.org/scripts/script.php?script_id=184

http://www.vim.org/scripts/script.php?script_id=69

http://vim-taglist.sourceforge.net

Auditing Java code, or IntelliJ IDEA as poor man's Fortify

This Twitter exchange got me to look at IDEA. The below are a few tips and config files that can get you started with using IDEA for manual audits. IMHO the results will be close to what you get from commercial code scanner, albeit slightly slower :)

IntelliJ features useful for code auditing

1. "Analyze" ->"Data flow to here" or "Data flow from here" https://www.jetbrains.com/idea/webhelp/analyzing-data-flow.html
2. "Analyze" -> "Inspect code"(next section below)
3. All kinds of source navigation - to definition, to usages etc. Shortcuts below.

Custom code inspections

One of good starting points for basic concepts of code audits is Chapter 19 of "Web Application Hacker's Handbook" 2ed. You obviously have to have a clue what this security thing is all about, and this post is not an audit tutorial.

I've created some custom inspections for IDEA, as its original "Security" set is quite arbitrary and if it targets anything, it is applets, not web apps. My inspection policies are based on the WAHH2E book and are geared towards identifying user input, dodgy file operations and so on.

You can get the policies from https://github.com/agelastic/intellij-code-audit

Installing the inspection policies

Open Options / Inspections. Import the policy you want to use (the full policy may produce a lot of results on very large projects, e.g. full Confluence produces about 2000 results, so I made partial policies as well), check "Share Profile", or you won't be able to use this policy.

Configuring custom code inspections in IDEA

Each of my inspection configs contains a single enabled item - "General"/"Structural Search Inspection" with a number of templates.

TIP: For some reason IntelliJ sometimes ignores the imported policy and as a result you will have no findings. What seems to work is to scan the code with any builtin policy, for example, "Default", and then run the security one. If you do not see the "Inspecting code..." progress window, then analysis did not happen.

Templates will find points in code that are interesting for security auditor - HTTP parameters, file access, sessions, SQL queries, etc. Then you can use data flow analysis (point 1 in the list above) or simply navigate through source (below).

Running an analysis

Open "Analyze" / "Inspect code", select the policy, scope etc, run, check results. There are various ways of marking things as reviewed - see "suppress" options in the results pane. They are the same as with any other alerts produced by code inspection engine.

It may be useful (depending on a kind of the finding) to investigate data flow to and from the found input/output point.

IntelliJ's docs for this feature, which in turn uses very powerful Structural Search & Replace engine are at https://www.jetbrains.com/idea/webhelp/creating-own-inspections.html

Code navigation shortcuts

These are collected from Stackoverflow posts. They are for Mac OS, Windows shortcuts usually have CTRL instead of CMD

Cmd + Shift + A - opens a window where you can search for GUI commands
Cmd + Alt + Left to get back, Cmd + Alt + Right  to “go forward”
Cmd + B - Go to declaration
Cmd + Alt + B - Go to implementation
Cmd + U - Go to super-method/super-class
Cmd + Alt + F7 - Show usages
Cmd + N - Go to class
Cmd + P - Parameter info
Cmd-F - Find, F3/Shift + F3 - Find next/previous, Ctrl + Shift + F - Find in path
F2/Shift + F2 - Next/previous highlighted error
Cmd + E - recently used files
Cmd + P - Parameter info
Ctrl + Shift + Q - Context info
Ctrl + J - Quick documentation

Learning shortcuts

There is a great plugin that will teach you shortcuts fast - Key Promoter. Any time you do something with menus, it will show you what shortcut you could use to achieve the same effect.

Bonus: Code Reviewing Web App Framework Based Applications