186 lines
7.5 KiB
Plaintext
186 lines
7.5 KiB
Plaintext
|
===========================================================================
|
||
|
|
Using physical DMA provided by OHCI-1394 FireWire controllers for debugging
|
||
|
|
===========================================================================
|
||
|
|
|
||
|
|
Introduction
|
||
|
|
------------
|
||
|
|
|
||
|
|
Basically all FireWire controllers which are in use today are compliant
|
||
|
|
to the OHCI-1394 specification which defines the controller to be a PCI
|
||
|
|
bus master which uses DMA to offload data transfers from the CPU and has
|
||
|
|
a "Physical Response Unit" which executes specific requests by employing
|
||
|
|
PCI-Bus master DMA after applying filters defined by the OHCI-1394 driver.
|
||
|
|
|
||
|
|
Once properly configured, remote machines can send these requests to
|
||
|
|
ask the OHCI-1394 controller to perform read and write requests on
|
||
|
|
physical system memory and, for read requests, send the result of
|
||
|
|
the physical memory read back to the requester.
|
||
|
|
|
||
|
|
With that, it is possible to debug issues by reading interesting memory
|
||
|
|
locations such as buffers like the printk buffer or the process table.
|
||
|
|
|
||
|
|
Retrieving a full system memory dump is also possible over the FireWire,
|
||
|
|
using data transfer rates in the order of 10MB/s or more.
|
||
|
|
|
||
|
|
With most FireWire controllers, memory access is limited to the low 4 GB
|
||
|
|
of physical address space. This can be a problem on IA64 machines where
|
||
|
|
memory is located mostly above that limit, but it is rarely a problem on
|
||
|
|
more common hardware such as x86, x86-64 and PowerPC.
|
||
|
|
|
||
|
|
At least LSI FW643e and FW643e2 controllers are known to support access to
|
||
|
|
physical addresses above 4 GB, but this feature is currently not enabled by
|
||
|
|
Linux.
|
||
|
|
|
||
|
|
Together with a early initialization of the OHCI-1394 controller for debugging,
|
||
|
|
this facility proved most useful for examining long debugs logs in the printk
|
||
|
|
buffer on to debug early boot problems in areas like ACPI where the system
|
||
|
|
fails to boot and other means for debugging (serial port) are either not
|
||
|
|
available (notebooks) or too slow for extensive debug information (like ACPI).
|
||
|
|
|
||
|
|
Drivers
|
||
|
|
-------
|
||
|
|
|
||
|
|
The firewire-ohci driver in drivers/firewire uses filtered physical
|
||
|
|
DMA by default, which is more secure but not suitable for remote debugging.
|
||
|
|
Pass the remote_dma=1 parameter to the driver to get unfiltered physical DMA.
|
||
|
|
|
||
|
|
Because the firewire-ohci driver depends on the PCI enumeration to be
|
||
|
|
completed, an initialization routine which runs pretty early has been
|
||
|
|
implemented for x86. This routine runs long before console_init() can be
|
||
|
|
called, i.e. before the printk buffer appears on the console.
|
||
|
|
|
||
|
|
To activate it, enable CONFIG_PROVIDE_OHCI1394_DMA_INIT (Kernel hacking menu:
|
||
|
|
Remote debugging over FireWire early on boot) and pass the parameter
|
||
|
|
"ohci1394_dma=early" to the recompiled kernel on boot.
|
||
|
|
|
||
|
|
Tools
|
||
|
|
-----
|
||
|
|
|
||
|
|
firescope - Originally developed by Benjamin Herrenschmidt, Andi Kleen ported
|
||
|
|
it from PowerPC to x86 and x86_64 and added functionality, firescope can now
|
||
|
|
be used to view the printk buffer of a remote machine, even with live update.
|
||
|
|
|
||
|
|
Bernhard Kaindl enhanced firescope to support accessing 64-bit machines
|
||
|
|
from 32-bit firescope and vice versa:
|
||
|
|
- http://v3.sk/~lkundrak/firescope/
|
||
|
|
|
||
|
|
and he implemented fast system dump (alpha version - read README.txt):
|
||
|
|
- http://halobates.de/firewire/firedump-0.1.tar.bz2
|
||
|
|
|
||
|
|
There is also a gdb proxy for firewire which allows to use gdb to access
|
||
|
|
data which can be referenced from symbols found by gdb in vmlinux:
|
||
|
|
- http://halobates.de/firewire/fireproxy-0.33.tar.bz2
|
||
|
|
|
||
|
|
The latest version of this gdb proxy (fireproxy-0.34) can communicate (not
|
||
|
|
yet stable) with kgdb over an memory-based communication module (kgdbom).
|
||
|
|
|
||
|
|
Getting Started
|
||
|
|
---------------
|
||
|
|
|
||
|
|
The OHCI-1394 specification regulates that the OHCI-1394 controller must
|
||
|
|
disable all physical DMA on each bus reset.
|
||
|
|
|
||
|
|
This means that if you want to debug an issue in a system state where
|
||
|
|
interrupts are disabled and where no polling of the OHCI-1394 controller
|
||
|
|
for bus resets takes place, you have to establish any FireWire cable
|
||
|
|
connections and fully initialize all FireWire hardware __before__ the
|
||
|
|
system enters such state.
|
||
|
|
|
||
|
|
Step-by-step instructions for using firescope with early OHCI initialization:
|
||
|
|
|
||
|
|
1) Verify that your hardware is supported:
|
||
|
|
|
||
|
|
Load the firewire-ohci module and check your kernel logs.
|
||
|
|
You should see a line similar to::
|
||
|
|
|
||
|
|
firewire_ohci 0000:15:00.1: added OHCI v1.0 device as card 2, 4 IR + 4 IT
|
||
|
|
... contexts, quirks 0x11
|
||
|
|
|
||
|
|
when loading the driver. If you have no supported controller, many PCI,
|
||
|
|
CardBus and even some Express cards which are fully compliant to OHCI-1394
|
||
|
|
specification are available. If it requires no driver for Windows operating
|
||
|
|
systems, it most likely is. Only specialized shops have cards which are not
|
||
|
|
compliant, they are based on TI PCILynx chips and require drivers for Windows
|
||
|
|
operating systems.
|
||
|
|
|
||
|
|
The mentioned kernel log message contains the string "physUB" if the
|
||
|
|
controller implements a writable Physical Upper Bound register. This is
|
||
|
|
required for physical DMA above 4 GB (but not utilized by Linux yet).
|
||
|
|
|
||
|
|
2) Establish a working FireWire cable connection:
|
||
|
|
|
||
|
|
Any FireWire cable, as long at it provides electrically and mechanically
|
||
|
|
stable connection and has matching connectors (there are small 4-pin and
|
||
|
|
large 6-pin FireWire ports) will do.
|
||
|
|
|
||
|
|
If an driver is running on both machines you should see a line like::
|
||
|
|
|
||
|
|
firewire_core 0000:15:00.1: created device fw1: GUID 00061b0020105917, S400
|
||
|
|
|
||
|
|
on both machines in the kernel log when the cable is plugged in
|
||
|
|
and connects the two machines.
|
||
|
|
|
||
|
|
3) Test physical DMA using firescope:
|
||
|
|
|
||
|
|
On the debug host, make sure that /dev/fw* is accessible,
|
||
|
|
then start firescope::
|
||
|
|
|
||
|
|
$ firescope
|
||
|
|
Port 0 (/dev/fw1) opened, 2 nodes detected
|
||
|
|
|
||
|
|
FireScope
|
||
|
|
---------
|
||
|
|
Target : <unspecified>
|
||
|
|
Gen : 1
|
||
|
|
[Ctrl-T] choose target
|
||
|
|
[Ctrl-H] this menu
|
||
|
|
[Ctrl-Q] quit
|
||
|
|
|
||
|
|
------> Press Ctrl-T now, the output should be similar to:
|
||
|
|
|
||
|
|
2 nodes available, local node is: 0
|
||
|
|
0: ffc0, uuid: 00000000 00000000 [LOCAL]
|
||
|
|
1: ffc1, uuid: 00279000 ba4bb801
|
||
|
|
|
||
|
|
Besides the [LOCAL] node, it must show another node without error message.
|
||
|
|
|
||
|
|
4) Prepare for debugging with early OHCI-1394 initialization:
|
||
|
|
|
||
|
|
4.1) Kernel compilation and installation on debug target
|
||
|
|
|
||
|
|
Compile the kernel to be debugged with CONFIG_PROVIDE_OHCI1394_DMA_INIT
|
||
|
|
(Kernel hacking: Provide code for enabling DMA over FireWire early on boot)
|
||
|
|
enabled and install it on the machine to be debugged (debug target).
|
||
|
|
|
||
|
|
4.2) Transfer the System.map of the debugged kernel to the debug host
|
||
|
|
|
||
|
|
Copy the System.map of the kernel be debugged to the debug host (the host
|
||
|
|
which is connected to the debugged machine over the FireWire cable).
|
||
|
|
|
||
|
|
5) Retrieving the printk buffer contents:
|
||
|
|
|
||
|
|
With the FireWire cable connected, the OHCI-1394 driver on the debugging
|
||
|
|
host loaded, reboot the debugged machine, booting the kernel which has
|
||
|
|
CONFIG_PROVIDE_OHCI1394_DMA_INIT enabled, with the option ohci1394_dma=early.
|
||
|
|
|
||
|
|
Then, on the debugging host, run firescope, for example by using -A::
|
||
|
|
|
||
|
|
firescope -A System.map-of-debug-target-kernel
|
||
|
|
|
||
|
|
Note: -A automatically attaches to the first non-local node. It only works
|
||
|
|
reliably if only connected two machines are connected using FireWire.
|
||
|
|
|
||
|
|
After having attached to the debug target, press Ctrl-D to view the
|
||
|
|
complete printk buffer or Ctrl-U to enter auto update mode and get an
|
||
|
|
updated live view of recent kernel messages logged on the debug target.
|
||
|
|
|
||
|
|
Call "firescope -h" to get more information on firescope's options.
|
||
|
|
|
||
|
|
Notes
|
||
|
|
-----
|
||
|
|
|
||
|
|
Documentation and specifications: http://halobates.de/firewire/
|
||
|
|
|
||
|
|
FireWire is a trademark of Apple Inc. - for more information please refer to:
|
||
|
|
https://en.wikipedia.org/wiki/FireWire
|