lost telnet & ftp upon upgrade to 2.1.0.0A_201703071456 firmware

[andy2301]

#My 1080p v1 comes with firmware 2.0 (2.0.0.1A_201612051401). I was able to apply this hack to get telnet and ftp. Thanks to @xmflsct!

However, yesterday I upgraded (through the YiHome app) to the latest 2.1 (2.1.0.0A_201703071456) firmware, and the hack doesn't work any more. Yes i tried to reapply the hack from scratch. I lost telnet & ftp.

Also tried to store the firmware using the home_y20m dump generated by @xmflsct as posted here, but it didn't restore at all (renamed it to "home" and put at root of sd card). I suspect it's a generated home partition dump, not an official firmware.

I'd appreciate if anybody can share a copy of the official firmware 2.0.0.1A_201612051401 for 1080p v1.

Btw, you can also download the latest firmware 2.1.0.0A_201703071456 here.
http://www.xiaoyi.com/home/firmware/download/?version=4
Click on "download" of the "2.1.0.0A_201703071456" firmware

[xmflsct]

I did not even dare to upgrade myself. :D So I have no idea what has been changed in the newer upgrade.

But feel free to use my copy, or other copies provided in that thread.

[andy2301]

@xmflsct I tried to restore your copy through the "manual firmware upgrade" approach, but it didn't do anything. I'm still with the upgraded 2.1 version. Do you know another way to apply/use your copy?

My guess is that the new firmware gets rid of the factory_test.sh hole in its init.sh file. I tried to look into the firmware through binwalk but it appears the firmware is encrypted (binwalk returns nothing).

[shadow-1]

@andy2301
Yes you are absolutely correct. I have confirmed with the Yi Dome cameras that firmware upgrades remove /home/app/script/factory_test.sh (which in turn calls the script from microSD card). Hence why my own custom firmware goes in a different direction.

I can create a version of my custom firmware for this camera. However I can't test it as I don't have this camera. You MUST have access to the terminal through the serial port for troubleshooting.

As mentioned elsewhere. Official firmware upgrade file is not really a firmware file. It is an encrypted archive. Explanation and instructions on how to unpack below.
fritz-smh#124 (comment)

[andy2301]

Great explanation of the firmware! @shadow-1
I'll get a serial port later tonight for troubleshooting.

[xmflsct]

Hello again guys. I tried to reflash my home partition with @shadow-1's copy and my copy, both doesn't work somehow. I remember I used it when I restored my camera last time when I bricked it.. Now I have detached the serial connection just today... Tomorrow I will attach them again and see what happened, and try to produce a new flashable image.

[xmflsct]

Oops, my copy is actually working again somehow. @andy2301 Do you still have it, and would you like to try again?
My yellow flashes slowly when flashing, in between it flashes very quick for a second then comes back. After flashing is done, I have access to my cam again luckily.

[andy2301]

How did you flash it?

When I tried it this morning, I did the following:
I downloaded the zip file you posted, extracted to the root of sd card, rename it to "home". But it did't do anything for me. Upon reboot, it's still my upgraded 2.1 version.

[shadow-1]

@andy2301
Do not rename the file. It needs to be named home_y20.
In addition, make sure your microSD card is FAT32 formatted. By default high capacity SD cards (over 32gb) are formatted using exFAT.

[xmflsct]

@andy2301 Aha, that's why. Because both @shadow-1's and mine copy are for uboot to use, which means that it differs from the official way of updating it. So as said, you should keep it as home_y20.

Very interesting thing is that. With just @shadow-1's copy, it does not work (I have no serial output to examine the problem now). After using @shadow-1's copy, I use mine again which gives a bit different yellow led flashing pattern and that worked - at least to bring the brick device back.

[shadow-1]

@xmflsct
U-Boot does a check to see whether the recovery image matches what is currently on the system. If it thinks they are the same, the firmware upgrade is skipped. Considering it is U-Boot that is doing the check, I believe the check must be quite primitive.

From experimentation (I have flashed my camera probably over 50 times debricking and getting the region ban workaround working), it appears that the check to see whether the firmware on the system matches the recovery image is based on the embedded name within the image and the timestamp. So most likely, your current recovery image will no longer work as it matches what is currently on the system. However my one will work now because (most likely) the embedded name is different and the timestamp is definitely different.

[xmflsct]

@shadow-1
Good one! I need more debricking to catch up with your experience. :D

So I assume U-Boot thinks your image should be flashed, though it (might?) contain some error. After then I use my long-ago image, U-Boot now believes my image should be flashed.
Anyway, my long-ago image was simply wrapped the dd output with an uImage header, not as smart as your solution yet. I still need to figure out how I can properly pack jffs2 images. I think your v3 approach should be the way to go, as XiaoYi is tightening rules against hacking.

[shadow-1]

@xmflsct
My image is unlikely to contain an error (although it might). I created a fresh jffs2 filesystem with the files from your long-ago backup. You are restoring the entire partition including the free space whilst my one skips the free space. Hence my image is smaller and a little quicker to flash. The end result is exactly the same as we are restoring exactly the same files. The only difference is the uImage header will be slightly different.

[xmflsct]

@shadow-1 Yes, I fully understand this part. :) Will first follow your approach and update this project, then see how far I can go with RTSP support as this is the function that I need - I don't care if I can use it with app or not. I have another 720p one working right now as a security cam, and would like to have another one. :)

[shadow-1]

@xmflsct
Tomorrow I can send you an experimental version of my v3 firmware for this camera with working workaround for the region ban. However it will involve flashing the rootfs partition. As mentioned in the other thread, you have to upgrade uClibc which is almost impossible to upgrade on a running system on these cameras. Even if it were possible, it is highly risky and it is likely to brick. Using the offline upgrade through U-Boot is a much safer option.

I am confident that you will be able to restore your rootfs partition in case the camera gets bricked. The recovery method is exactly the same as for the home partition.

[xmflsct]

Thank you so much in advance! I will have a try of your ones and let you know if it is working.

Saying that, I have a 040 version of SDK. I think they should have 050 by now. I will share it if I can find it and download it from somewhere.

[shadow-1]

Thanks for the updated SDK. I knew the 040 version has been released, however I didn't find a download link for it yet. When I got the 030 version, I don't think the 040 version was released (perhaps it was very new). I haven't heard anything about a 050 version yet.

With a little luck HiSilicon may have provided an example RTSP server with the SDK. Unlikely...but we can always be hopeful.

[xmflsct]

You are welcome. :) Not sure what had been updated though. Hope it would be a bit useful to you. 050 is not released yet, at least not on the forum where I downloaded 040. While browsing, I found a post with an updated sample code that claims to support RTSP for Hi3518EV200. I will have a try tomorrow, see if I can compile it correctly. Otherwise I might need your help again. :)

[andy2301]

@xmflsct Using your copy of home_y20, I've got my camera back to version 2.0.0.1A_201612051401. That's awesome!

Just extract it under the root directory of sd card and reboot. Make sure the name is "home_y20".

[xmflsct]

@andy2301 Glad that it works!

Do you happen to have serial connection as well? If so, can you try to change the boot delay and see if you can enter U-Boot fritz-smh#141 (comment) My camera doesn't respond to any key press signal, so I was wondering whether they tweak the hardware or just a faulty part in my camera.

If you succeed in doing so, it would be nice if you could upgrade to the newest version again, and dump out the entire system in U-Boot. That provides an overview of what has been updated, and could be a base for the next step of this project. Let me know if this is feasible for you. :)

[shadow-1]

@xmflsct
No need to tweak the U-Boot environment as you know how to flash rootfs and home partitions now. It should be a simple change on the env partition to get the countdown working if you really want to.

There is no need to upgrade the firmware to the newest version and dump the entire filesystem. I have provided instructions on how to extract the firmware and the upgrade is simply a replacement of the files on the camera within the /home directory.

Even so, if you want to make a backup of the camera and have serial access. It is easiest to use dd once booted rather than mucking around with U-Boot.

[xmflsct]

@shadow-1
That's true. Then I will wait for your updated release, and combine it into this 1080p camera. :)

[shadow-1]

@xmflsct
I got around to creating a test firmware for the Yi 1080 Home. The links are below:
rootfs_y20
home_y20

Creating the home partition took longer than what I thought it would. I need to create some more build scripts to make it fully automated.

To make the upgrade more controlled and to make it easier for issues to be identified and rectified, what I recommend to do is follow the following steps:

1. Upgrade only the rootfs partition.
2. Boot up the camera and check that everything still works just as before (telnet working etc.) If there is an issue at this point, we will know that I deleted something I wasn't meant to and it's preventing bootup.
3. Flash the home partition.
4. Test whether telnet is working, FTP is working, web server is working (all services are enabled by default on the firmware).
5. Enter a valid proxychains configuration with at least one proxy server from Mainland China. This will allow your camera to not complain about the region ban and be fully operational with the Xiaomi smartphone app.

---

For others that may want to try the firmware, please ensure you have access to the terminal through the serial port for debugging purposes. I do not have this particular camera so the firmware is completely untested.

[xmflsct]

@shadow-1
Thank you so much! I will try it just now and let you know how it goes.

By the way, the code I mentioned yesterday, I tried to compile it again, and still get the annoying line 1: syntax error: unexpected word (expecting ")"). All posts on the internet suggest that I might output an object file, but it is actually not - I test it with file which shows it as an executable. I suspect there might be some wrong flags in SDK's Makefile.param but my knowledge is not enough to find it out.
Would you like to have a try? Here is the archive.
hp-ipc_rtsp.zip

[maysamsh]

@shadow-1 I tried the rootfs, it revived the cam! When I tried to pair it with the phone it told me "This camera can only used within China", then I tried the other fine, home_y20, but unfortunately it didn't work and once again the status is still yellow.

[shadow-1]

@maysamsh
Congratulations, it looks like your problem was with the rootfs partition afterall and it has now been restored.

This is completely experimental firmware and I have done quite a lot of changes to the home partition. I may have made a mistake with it because of all the changes I made without testing. You can restore the original home partition. You can use @xmflsct copy or my one and your camera will be back to how it was.

[xmflsct]

@maysamsh Yes, with my own copy of home_y20, I get my cam back and I can use it as before.

@shadow-1 Same happened to me. Flashing rootfs is fine, but bricked after home_y20. Still, I cannot run the proxychains4 from your home_h20 extract. I copy both the executable and the .so to exact the same location on my cam, I get Illegal instruction. The ld-uClibc-0.9.33.2.so in my /lib seems to be updated already with a last modify date at 03/06/17. Is that correct?

[maysamsh]

@shadow-1 Unfortunately it does not work anymore, even I have tried restoring images you already provided. What was the magic?!

[shadow-1]

@xmflsct
Ah ok...looks like I tried to do too much at once.

I manually updated the firmware to 2.1.0.0A_201703071456.
I must have messed it up (I did it quite quickly), I'll redo to firmware upgrade with 2.0.0.1A_201612051401 firmware which we know works properly.

Yes I updated ld-uClibc-0.9.33.2.so already from a previous compile. You will also notice that Busybox on rootfs has also updated to include a few extra but useful utilities.

[shadow-1]

@xmflsct
I am very confused why you get Illegal instruction error with those programs.
Can you provide me with the output of:
cat /proc/cpuinfo

[xmflsct]

@shadow-1
No worries. I think we are progressing very fast already. :)
Sadly my limited knowledge to C/C++ cannot contribute that much of you guys' programming. But I am up for any testing/debugging/researching/etc./etc., to save your time!

[vaidyasr]

@andy2301
Link for 2.0.0.1A_201612051401
http://download.xiaoyi.com/smarthomecam/2.0.0.1A_201612051401home_y20m

[andy2301]

Thanks for providing the firmware @vaidyasr .

Guys, i cooked another version of rtsp server that streams from /tmp/view continuously. This one should work side by side with the existing stock features. So far it's only tested on the original 2.0.0.1A firmware.
rtsp2303_live_bin_20170414-1630.zip

[andy2301]

Remember that my prior streaming solution rtsp2301 requires to shut down the stock streaming process rmm, rendering the YiHome app function and all other motion detection functionalities unavailable. So, I started looking for a different solution that co-exists with the stock features.

Thanks to @shadow-1, who mentioned that the rmm process writes two streams 1080p and 360p to file /tmp/view with 10 seconds of video data. So this rtsp2303 solution focuses on streaming from /tmp/view file (1586752 bytes).

Think of the /tmp/view as a loop file that the rmm process writes to continuously (start over once it reaches the end-of-file offset 1586752. So, in order to stream from this file, we need to solve the following problems:

1. find out the file offset boundary of the 1080p stream.
   I've experimented a lot and it seems to me that offset range [0, 1048576] (the first 1MB) is a safe range for 1080p stream. Sometimes [0, 1152000] gives you more frames of 1080p video, but other times it gives you the 360p video. So in this program I sticked to the safe range of [0, 1048576] or [0, 1050000].
2. find out which part of the offset range [0, 1048576] to start, in order to reduce the lagging issue.
   If you start from offset 0, your lag is random, and could range from [1-10] seconds. I did implement a function "monitorChanges" to detect (at the beginning of streaming or could be set to run periodically) the latest changes and start streaming from there.
   It works to reduce the lag most of time to [2-4] seconds. But I couldn't get it better than that. Once you get too close, you will get garbled pictures, probably because the writing of latest 1080p data is still in flux to file.
   Anyhow, you are welcome to try it out by yourself with the env variable settings (adjIDX), as demonstrated in the rtsp2303.sh script.

In summary:
Pros:

• Co-exist with all the stock features
• Much less overhead. The cpu usage is only 12% while streaming 1080p at 15fps (the same frame rate written to /tmp/view and the recorded .mp4 files).

Cons:

• The streamed video lags behind [2-4] seconds most of time, but sometimes it could lag up to 10 seconds.
• The server is stable but the client app might exit or stop streaming. After some time (30 seconds ~ 4 minutes observed from my experiments) the streaming client app may stop. The reason, I suspect, is that the PCR timestamp in the /tmp/view confuses the client app and it simply can't handle any more after some time.

Let me know if anybody has better approaches to solve the 2 problems:

• the exact offset range of the 1080p stream in /tmp/view
• where to start reading from the file at any given time, in order to reduce the lagging

[andy2301]

There are some of the variables used in rtsp2303. You can change these default values from the command line by setting env variables (as demonstrated in rtsp2303.sh script).

unsigned long startOFS = 0;          // the starting file offset to read from /tmp/view
unsigned long repeatOFS = 1050000;   // we rewind the file to the starting file offset when it reaches this repeatOFS. Basically we only read [start, repeat].
unsigned int numCHUNK = 7;          // we split the [start, repeat] offset range of /tmp/view into numCHUNK chunks, in order to determine which chunk has changed
unsigned long sleepMS = 500;       // read a buf from each chunk of /tmp/view, sleep for sleepMS, and read another buf to compare which chunk changed
unsigned long cmpSZ = 10240;       // the number of bytes of data to read from each chunk of /tmp/view
int adjIDX = -1;               // if we detect that chunk[5] changed, we may want to start reading from chunk[5 + adjIDX]
int readsPMON = 0;      // reads per monitor: how many doReadFromFile invokes (file reads) before we run each monitorChanges.  0 = disable monitorChanges
int VERB = 0;                 // print out more info if set to 1

The following is the logic of the monitorChanges implementation.

Monitor which part of file /tmp/view just got changed, so that we can read from there to get the latest video stream. The purpose is to make sure we stream the latest video segments rather than possibly 10 seconds lagging behind (file /tmp/view stores roughly 10 sec of video).

1. divide file offset range [startOFS, repeatOFS] of /tmp/view into num_chunks chunks (logically)
2. take a snapshot of cmpSZ bytes of data from each chunk [chunkStartOfs, chunkStartOfs+cmpSZ]
3. sleep for sleepMS mili-seconds
4. take another snapshot of cmpSZ bytes of data from each chunk [chunkStartOfs, chunkStartOfs+cmpSZ]
5. for each chunk, compare the two snapshots for changes. Mark the chunk as changed if any.
6. return the chunkStartOfs offset of the chunk that just got changed if adjIDX=0, otherwise, return the offset of chunk id (changedChunkIdx + adjIDX).
   If adjIDX = -1, return the offset of the prior chunk. This is the default. The rationale is if we read the current changing chunk, we might get fragmented frame/data.
   If adjIDX = -2, return the offset of the prior chunk. This is safer to avoid fragmented frame/data but may have increased lagging.
   If adjIDX = 1, return the offset of the next chunk.
   .....

[shadow-1]

@andy2301
Great work! I won't have a lot of time to test this yet.
However I do have an observation:

Are you sure /tmp/view is a circular buffer? From my experimentation with the live555 library in the past, I never seen behaviour that suggests that it could be a circular buffer. From my experimentation, I was led to believe that it is probably a simple FIFO buffer, so the latest video data is located in the same area in the file. So the video that is roughly 10 seconds behind live feed is always towards the start of the file whilst the closest to 'live' video is towards the end of the stream in /tmp/view.

Unfortunately I do not know the offset between the 1080p and the 360p streams.

[andy2301]

@shadow-1 I think it's a circular buffer.
Following your post, I ran the following to always stream the 150k bytes [900000, 1050000]. It simply repeats the 1~2 seconds of video frames, for example 18:12:41 - 42, and then after 10 seconds it jumps forward to 18:12:50 - 51.

startOFS=900000 repeatOFS=1050000 VERB=1 readsPMON=-1 ./rtsp2303

[shadow-1]

@andy2301
Ah ok, no problems.

With regards to the offsets, I have a suspicion that it will be either located somewhere within /tmp/mmap.info or be hardcoded within the application.

When I get more free time, I can disassemble the Xiaomi applications and trace what is being read from /tmp/mmap.info.

[andy2301]

Please do when you get time. Thanks. @shadow-1

[xmflsct]

@andy2301
Wonderful progress! I am away currently, and will be back for helping testing in three days.

[xmflsct]

@andy2301 I am back, and just test a bit. Indeed after every reboot, rtsp2303 performs differently as per chunk data varies. mmap.info also varies a bit after each reboot, but I am not sure if those differences represent chunk size or offset. But at least in the v1 example, mmap.info is used somehow.

To provide a separate idea that might be useful. Since when streaming with live555 of /tmp/view, it performs rather stable of each connection. Maybe, we could use h264 headers to determine each chunk/frame to adapt to the file dynamically? I will look into if I can analyse /tmp/view from h264 header perspective and update if I find something useful.
http://stackoverflow.com/questions/6394874/fetching-the-dimensions-of-a-h264video-stream

[andy2301]

Sorry for late reply. I went on a trip and just got back.

I didn't get much useful info from mmap.info. Maybe @shadow-1 can help us there.

@xmflsct yes it'll be great if you can figure out how to read the h264 headers and adapt to the file dynamically. rtsp2303 is based on live555. I'm attaching the source code. My modification involves mainly rtsp2303.cpp and livemedia/ByteStreamFileSource.cpp. You can do a global search of "andy2301Mod" for other changes (e.g. FrameRates changed from 25 to 15).
rtsp2303_srcbin_20170414-1630.zip

[xmflsct]

@andy2301 No worries. Hope you have had a great time. :)

Recently I was also thinking about using live555 as a base, since the test program can stream /tmp/view without problem, and we just have to somehow tell it to restart every x seconds. By using live555, I assume we don't need to find h264 headers ourselves anymore - it should be taken care of by live555.
I will take a look into your source code.

(UPDATE)
I read through what you have modified. It seems like you are trying to actually tell live555 which chunk of data is one frame. Isn't that not necessary? I keep your rtsp2303.cpp with original ByteStreamFileSource.cpp, the compiled program runs as the test program does - stops after streaming both 1080p and 360p videos. Would that be an idea just to restart per x second and let live555 takes care of the rest?

[shadow-1]

@andy2301 @xmflsct
I hope to find some time to start working on RTSP soon!

I like Live555 because it is a very mature library that is actually used in many commercial applications. However what I don't like about it is that it is written in C++ which makes the file size quite large (in comparison with C programs) and of course there is a dependency on the standard C++ library libstdc++.

@xmflsct
I think you are right, I think we should be working out where to start streaming from and then continue streaming from there on whilst rolling back to the start of the file when it reaches an offset. Easier said than done of course.

@andy2301
Did you compile your own version of libstdc++?

What really put me off C++ applications on these cameras is that libstdc++ that comes with the SDK is almost 5mb in size! That is absolutely ridiculously large which makes C++ programs completely unsuitable for these cameras as they have 16mb of flash memory in total.

However I noticed that libstdc++ that you included in your source file is under 1mb in size. Much more manageable.

[xmflsct]

@shadow-1
That's wonderful! I will rely on you guys to take RTSP service further. :)

[andy2301]

The libstdc++ lib is from the Yi 720p v1 firmware. I used a stripper called upx which reduced the rtsp2303 footprint to less tha 200k.

[xmflsct]

@andy2301 I have put my 1080p camera into production together with my other original 720p one. It runs now with your rtsp2301 process. It sometimes crashes so I add a watch process to restart it when this happens. Luckily with current setup, I can easily update files remotely and test new setups. Please let me know if you have further progress with rtsp2303. :)

[naterh]

@shadow-1 late to the party but, looks like there are some http streaming add-ons included with the SDK

http://blog.51cto.com/shugenyin/1851241

[saundersrg]

You people are legends. I've got RTSP running on a Yi Dome 1080p with @shadow-1 firmware and @andy2301 script, thank you!

[gelsas]

@saundersrg could you share the final files you used and a short tutorial? I would love to use it aswell on my dome 1080p

[saundersrg]

It really didn't work as well as I'd hoped tbh, and was really simple. I've ended up selling the Yi and buying a Foscam FI9821P. Comparable pricing with full ONVIF, RTSP and support in Zoneminder.

Firstly I flashed with the firmware from @shadow-1, and made sure I could SSH. I then took the scripts from @andy2301 moved them (WinSCP/PuTTY) into a folder under /home/ and tested that I could launch manually. The feed would work for about 10s and then the client device would disconnect (be it VLC or ZM), and I'd have to keep enabling and disabling the monitor in ZM. For some reason I couldn't get Andy's script to launch automatically at any stage, using rc or the init scripts.

[andy2301]

@saundersrg the client device disconnecting issue is expected, as I mentioned in my previous comment when I published rtsp2303. Recently I got some help from another thread. Might be able to fix this if things go well.

[squirrel289]

Hey @andy2301! Thanks for sharing your work with everyone. It seems that, between your contributions and those of the several base images (like this one and @shadow-1) many have nearly managed to get a working rtsp solution on these puppies. I wanted to see if you'd managed to incorporate the latest changes you referred to in your last note. If so, any chance you'd be willing to share? If not, would you be willing to point others to the same advice and to publish your current work to a public repo so others can fork and/or create pull requests?

Thanks,
BS