logo

Could not compile stylesheet for simplistic. Using last compiled stylesheet.

Wilink8 panic and hardware issues in dmesg

posted in IGEPv2
Wednesday, May 18 2016, 03:31 PM
0
Hello,
On some IGEPv2-RF boards, it is not possible to connect on configured WiFi AP with hostapd, and following some retries, it ends on issues in dmesg like panic and hardware errors (full dmesg log attached).

...
[ 1277.295623] wlcore: ERROR Tx stuck (in FW) for 5000 ms. Starting recovery
[ 1277.302703] ------------[ cut here ]------------
[ 1277.307586] WARNING: at /home/bnext/kernel/2.6.37-11-w8/compat-wireless-wilink8-0/drivers/net/wireless/ti/wlcore/main.c:818 wl12xx_queue_recovery_work+0x20/0x68 [wlcore]()
[ 1277.323516] Modules linked in: ppp_deflate ppp_async ppp_generic slhc ip_tables rfcomm hidp l2cap bluetooth wl18xx wlcore mac80211 option usb_wwan cfg80211 mcp251x ads7846 can_dev spidev omap_wdt wlcore_sdio usbserial compat cdc_acm
[ 1277.345123] [] (unwind_backtrace+0x0/0x118) from [] (warn_slowpath_common+0x4c/0x64)
[ 1277.355010] [] (warn_slowpath_common+0x4c/0x64) from [] (warn_slowpath_null+0x18/0x1c)
[ 1277.365081] [] (warn_slowpath_null+0x18/0x1c) from [] (wl12xx_queue_recovery_work+0x20/0x68 [wlcore])
[ 1277.376556] [] (wl12xx_queue_recovery_work+0x20/0x68 [wlcore]) from [] (wl12xx_tx_watchdog_work+0x130/0x154 [wlcore])
[ 1277.389465] [] (wl12xx_tx_watchdog_work+0x130/0x154 [wlcore]) from [] (process_one_work+0x1e0/0x32c)
[ 1277.400787] [] (process_one_work+0x1e0/0x32c) from [] (worker_thread+0x1ac/0x2dc)
[ 1277.410400] [] (worker_thread+0x1ac/0x2dc) from [] (kthread+0x7c/0x84)
[ 1277.419036] [] (kthread+0x7c/0x84) from [] (kernel_thread_exit+0x0/0x8)
[ 1277.427734] ---[ end trace 2648c1067299ab53 ]---
[ 1277.432556] wlcore: Reading FW panic log
[ 1277.437591] wlcore: Hardware recovery in progress. FW ver: Rev 8.6.0.0.5
[ 1277.446075] wlcore: pc: 0x0, hint_sts: 0x00000000 count: 1
[ 1277.452056] wlcore: down
[ 1277.455383] ieee80211 phy0: Hardware restart was requested
[ 1279.860443] wlcore: ERROR timeout waiting for the hardware to complete initialization
[ 1282.256408] wlcore: ERROR timeout waiting for the hardware to complete initialization
[ 1284.664489] wlcore: ERROR timeout waiting for the hardware to complete initialization
[ 1284.673431] wlcore: ERROR firmware boot failed despite 3 retries
[ 1284.679870] wlan0: failed to set key (1, ff:ff:ff:ff:ff:ff) to hardware (-11)
[ 1284.687316] wlan0: failed to set key (2, ff:ff:ff:ff:ff:ff) to hardware (-11)
[ 1829.879272] wlan0: failed to remove key (1, ff:ff:ff:ff:ff:ff) from hardware (-11)
[ 1829.887359] wlan0: failed to set key (1, ff:ff:ff:ff:ff:ff) to hardware (-11)

Here is the /etc/hostapd.conf file i set:
root@igep00x0:~# cat /etc/hostapd.conf
interface=wlan0
driver=nl80211
channel=6
skip_inactivity_poll=1
auth_algs=1
wpa=2
wpa_passphrase=########################
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CCMP
rsn_pairwise=CCMP
ssid=Observer-b912
bssid=08:00:28:00:B9:12
logger_syslog=-1
logger_syslog_level=0
logger_stdout=-1
logger_stdout_level=0

And the commands to activate wifi i use:
ifconfig wlan0 down
echo 0 > /sys/class/gpio/gpio139/value
echo 1 > /sys/class/gpio/gpio139/value
hostapd /etc/hostapd.conf &

With debug logs activated, i have very few messages displayed (on working well boards, i have a lot of messages)

Here are firmware versions (why two different versions?):
[ 629.627593] wlcore: PHY firmware version: Rev 8.2.0.0.168
[ 629.722656] wlcore: firmware booted (Rev 8.6.0.0.5)

Could you help me to understand and solve why wifi doesn't connect on some boards please?
(wifi chip problem? configuration problem? firmware version to update? wifi chip to calibrate? other issues?)

(I checked on TI Wilink8 that some calibration could be performed, if this could be the problem:
http://processors.wiki.ti.com/index.php/WL18xx
https://github.com/TI-OpenLink/18xx-ti-utils)

Thanks in advance for your help,
Simon
Responses (4)
  • Accepted Answer

    Monday, May 23 2016, 07:05 PM - #permalink
    0
    Hello,

    I tried to make some searches on the previous panic concerning Wilink8 module, and encountered the following forum on this topic:
    http://e2e.ti.com/support/wireless_connectivity/wilink_wifi_bluetooth/f/307/p/387252/1478675?pi239031348=2&pi239031349=2&pi239031350=1#pi239031350=1

    I checked links mentionned by Eduard in this forum, and i agree that the backport pointed by TI seems to concern R8-A6-02 release (and not above). Would it be a solution to try to backport newer wilink8 drivers? Have you tried this?
    On this other forum, it seems to be adviced to use porting for AM37x, would it be compliant and interesting to try it?
    https://e2e.ti.com/support/wireless_connectivity/wilink_wifi_bluetooth/f/307/t/148325

    Thanks in advance for your help.
    Best Regards,
    Simon
    The reply is currently minimized Show
  • Accepted Answer

    egavinc
    egavinc
    Offline
    Wednesday, May 25 2016, 07:03 PM - #permalink
    0
    Hello Simon, at the moment, the firmware that works with kernel 2.6.37 is the firmware that you are using.
    I have made several test with other versions of newest firmware without results.

    Some questions, Are you using the omap dsp in these boards? If you try to connect to a existing AP using wpa_supplicant, these boards can connect to it?

    Best Regards
    Eduard
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, May 26 2016, 04:46 PM - #permalink
    0
    Hello Eduard,

    Thanks a lot for your response and investigation on new firmware versions.

    Sorry for my low knowledge, i don't know so much about omap dsp (is it used for multimedia processing?).
    How could i check we are using it on our boards please? As i know, we don't explicitly use it or getting it to run.

    I tried to connect to an existing AP using wpa_supplicant with these two boards:
    - board1 where WiFi hostapd worked: wpa_supplicant connect on an existing AP (our WiFi Lan) and a DHCP IP address is assigned
    - board2 where WiFi hostapd failed: wpa_supplicant connect on an existing AP (our WiFi Lan) and DHCP IP is not assigned

    wpa_supplicant log (on both boards):
    root@igep00x0:~# wpa_supplicant -Dnl80211 -iwlan0 -c/etc/wpa_supplicant.conf -B -d
    Initializing interface 'wlan0' conf '/etc/wpa_supplicant.conf' driver 'nl80211' ctrl_interface 'N/A' bridge 'N/A'
    Configuration file '/etc/wpa_supplicant.conf' -> '/etc/wpa_supplicant.conf'
    Reading configuration file '/etc/wpa_supplicant.conf'
    Priority group 0
    id=0 ssid='Greenflex'
    netlink: Operstate: linkmode=1, operstate=5
    Own MAC address: de:ad:be:ef:00:00
    wpa_driver_nl80211_set_key: ifindex=4 alg=0 addr=0x4d1a0 key_idx=0 set_tx=0 seq_len=0 key_len=0
    wpa_driver_nl80211_set_key: ifindex=4 alg=0 addr=0x4d1a0 key_idx=1 set_tx=0 seq_len=0 key_len=0
    wpa_driver_nl80211_set_key: ifindex=4 alg=0 addr=0x4d1a0 key_idx=2 set_tx=0 seq_len=0 key_len=0
    wpa_driver_nl80211_set_key: ifindex=4 alg=0 addr=0x4d1a0 key_idx=3 set_tx=0 seq_len=0 key_len=0
    RSN: flushing PMKID list in the driver
    Setting scan request: 0 sec 100000 usec
    EAPOL: SUPP_PAE entering state DISCONNECTED
    EAPOL: Supplicant port status: Unauthorized
    EAPOL: KEY_RX entering state NO_KEY_RECEIVE
    EAPOL: SUPP_BE entering state INITIALIZE
    EAP: EAP entering state DISABLED
    EAPOL: Supplicant port status: Unauthorized
    EAPOL: Supplicant port status: Unauthorized
    Added interface wlan0
    Daemonize..


    DHCP connexion:
    board1:
    root@igep00x0:~# udhcpc -i wlan0
    udhcpc (v1.19.4) started
    Sending discover...
    Sending select for 192.168.70.95...
    Lease of 192.168.70.95 obtained, lease time 28800

    board2:
    root@igep00x0:~# udhcpc -i wlan0
    udhcpc (v1.19.4) started
    Sending discover...
    Sending discover...
    Sending discover...
    Sending discover...
    ...
    And there is this crash in dmesg:
    [ 52.468566] cfg80211: Calling CRDA to update world regulatory domain
    [ 53.061248] wlcore: ERROR ELP wakeup timeout!
    [ 53.065795] ------------[ cut here ]------------
    [ 53.070678] WARNING: at /home/bnext/kernel/2.6.37-11-w8/compat-wireless-wilink8-0/drivers/net/wireless/ti/wlcore/main.c:818 wl12xx_queue_recovery_work+0x20/0x68 [wlcore]()
    [ 53.086608] Modules linked in: ip_tables rfcomm hidp l2cap bluetooth wl18xx wlcore mac80211 cfg80211 option mcp251x usb_wwan spidev can_dev ads7846 omap_wdt wlcore_sdio compat usbserial cdc_acm
    [ 53.104644] [] (unwind_backtrace+0x0/0x118) from [] (warn_slowpath_common+0x4c/0x64)
    [ 53.114532] [] (warn_slowpath_common+0x4c/0x64) from [] (warn_slowpath_null+0x18/0x1c)
    [ 53.124633] [] (warn_slowpath_null+0x18/0x1c) from [] (wl12xx_queue_recovery_work+0x20/0x68 [wlcore])
    [ 53.136108] [] (wl12xx_queue_recovery_work+0x20/0x68 [wlcore]) from [] (wl1271_ps_elp_wakeup+0x18c/0x23c [wlcore])
    [ 53.148773] [] (wl1271_ps_elp_wakeup+0x18c/0x23c [wlcore]) from [] (wl1271_op_hw_scan+0x74/0xd4 [wlcore])
    [ 53.160675] [] (wl1271_op_hw_scan+0x74/0xd4 [wlcore]) from [] (__ieee80211_start_scan+0x254/0x320 [mac80211])
    [ 53.172912] [] (__ieee80211_start_scan+0x254/0x320 [mac80211]) from [] (ieee80211_request_scan+0x28/0x44 [mac80211])
    [ 53.185821] [] (ieee80211_request_scan+0x28/0x44 [mac80211]) from [] (nl80211_trigger_scan+0x480/0x4e4 [cfg80211])
    [ 53.198455] [] (nl80211_trigger_scan+0x480/0x4e4 [cfg80211]) from [] (genl_rcv_msg+0x1a8/0x1e8)
    [ 53.209350] [] (genl_rcv_msg+0x1a8/0x1e8) from [] (netlink_rcv_skb+0x50/0xac)
    [ 53.218597] [] (netlink_rcv_skb+0x50/0xac) from [] (genl_rcv+0x18/0x24)
    [ 53.227325] [] (genl_rcv+0x18/0x24) from [] (netlink_unicast+0x214/0x2ac)
    [ 53.236206] [] (netlink_unicast+0x214/0x2ac) from [] (netlink_sendmsg+0x294/0x324)
    [ 53.245910] [] (netlink_sendmsg+0x294/0x324) from [] (sock_sendmsg+0x90/0xac)
    [ 53.255157] [] (sock_sendmsg+0x90/0xac) from [] (sys_sendmsg+0x1ac/0x210)
    [ 53.264038] [] (sys_sendmsg+0x1ac/0x210) from [] (ret_fast_syscall+0x0/0x30)
    [ 53.273193] ---[ end trace 2648c1067299ab53 ]---
    [ 53.278900] wlcore: Reading FW panic log
    [ 56.080841] ------------[ cut here ]------------
    [ 56.085693] WARNING: at /home/bnext/kernel/2.6.37-11-w8/compat-wireless-wilink8-0/drivers/net/wireless/ti/wlcore/sdio.c:106 wl12xx_sdio_raw_read+0x12c/0x170 [wlcore_sdio]()
    [ 56.101715] Modules linked in: ip_tables rfcomm hidp l2cap bluetooth wl18xx wlcore mac80211 cfg80211 option mcp251x usb_wwan spidev can_dev ads7846 omap_wdt wlcore_sdio compat usbserial cdc_acm
    [ 56.119750] [] (unwind_backtrace+0x0/0x118) from [] (warn_slowpath_common+0x4c/0x64)
    [ 56.129638] [] (warn_slowpath_common+0x4c/0x64) from [] (warn_slowpath_null+0x18/0x1c)
    [ 56.139709] [] (warn_slowpath_null+0x18/0x1c) from [] (wl12xx_sdio_raw_read+0x12c/0x170 [wlcore_sdio])
    [ 56.151275] [] (wl12xx_sdio_raw_read+0x12c/0x170 [wlcore_sdio]) from [] (wl1271_recovery_work+0x1e8/0x484 [wlcore])
    [ 56.164001] [] (wl1271_recovery_work+0x1e8/0x484 [wlcore]) from [] (process_one_work+0x1e0/0x32c)
    [ 56.175079] [] (process_one_work+0x1e0/0x32c) from [] (worker_thread+0x1ac/0x2dc)
    [ 56.184692] [] (worker_thread+0x1ac/0x2dc) from [] (kthread+0x7c/0x84)
    [ 56.193298] [] (kthread+0x7c/0x84) from [] (kernel_thread_exit+0x0/0x8)
    [ 56.201995] ---[ end trace 2648c1067299ab54 ]---
    [ 56.206817] wl1271_sdio mmc1:0001:2: sdio read failed (-110)
    [ 56.212707] wlcore: Hardware recovery in progress. FW ver: Rev 8.6.0.0.5
    [ 56.219696] wlcore: down
    [ 56.223022] ieee80211 phy0: Hardware restart was requested
    [ 56.508178] wlcore: ERROR firmware boot failed despite 3 retries


    From this error, i found this forum: https://e2e.ti.com/support/wireless_connectivity/wilink_wifi_bluetooth/f/307/t/329596

    Could you help me to understand WiFi problems from this new informations please? I could execute, try other tests and provide all information you need.

    Thanks a lot for your help.

    Best Regards,
    Simon
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, May 31 2016, 11:44 AM - #permalink
    0
    Hi Eduard,

    I am still analyzing to try to understand why WiFi doesn't work on some IGEPv2 boards, and would really need help please.

    In my last tests, I executed hostapd command directly from a ssh terminal on 192.168.5.1 address, and keep the terminal active during the execution of the command: hostapd -dd /etc/hostapd.conf &

    On boards where WiFi works: there are directly hostapd activities when connecting on the IGEPv2 WiFi AP
    On boards where WiFi doesn't work: there are no hostapd activity when connecting on the IGEPv2 WiFi AP
    It could be observed that no nl80211 events is received in hostapd debug logs, like if WiFi RX wouldn't work.

    I would have the following investigation ideas (only guess):
    - would there be some power control activated concerning wifi modules? would there be some power on to activate?
    - i thought in calibration needs with calibrator tool, but it seems that wl18xx doesn't need it (unlike wl12xx)
    - how wilink 8 is connected in IGEPv2 schematics? does it use mmc? would it be possible that some IGEPv2 boards have hw routing problems concerning WiFi? (or Wifi rx)
    - would there be problems with wifi reception? antenna rx diversity?
    - would there be mac address problems? i set mac address using ifconfig wlan0 hw ether and bssid in hostapd.conf, but it seems that the nvs file already embed a mac address. calibrator tool could help to get this mac address. Could you provide me a cross-compiled calibrator tool for IGEPv2 please?

    Would you have feedback on these ideas please? would you think in something else that could cause problems i meet?

    I also may activate debug messages in modules, would you know the available procedures please?

    Thanks a lot if you could help me or give me a feedback (even small), i really don't know how to go forward on this problem now and get stuck.

    Best Regards,
    Simon
    The reply is currently minimized Show
Your Reply

SUPPORT


This email address is being protected from spambots. You need JavaScript enabled to view it.
This email address is being protected from spambots. You need JavaScript enabled to view it.
IGEP Community Wiki
IGEP Community Forum
IGEP Community Online Chat