Any advice for getting mipi-demo setup?

[benseleb]

Hi @sameer,

Firstly, thank you for doing all this work to get a working mipi-hdmi interface on the mkr vidor. I'm new to FPGA's and have been using this board to test some basic logic gate examples. Up until now, I've been using the method shown here to simultaneously upload SAMD21 code as well as the bit file to the FPGA.
I would love to implement and demo your code, and ultimately try to save image frames to an SD card. Having reviewed the files in your repository, I was wondering if the prior method could be used, or if another method is required?
Additionally, I am unfamiliar with HDLmake, is this tool also necessary for compiling?
I apologize for this basic implementation question, any help would be greatly appreciated.

Thanks!

[sameer]

Hey there, yeah I'm happy to help! I got busy with other projects and haven't gotten around to transferring images off-device.

I would love to implement and demo your code, and ultimately try to save image frames to an SD card. Having reviewed the files in your repository, I was wondering if the prior method could be used, or if another method is required?

Yeah that should work just fine. It's more a matter of writing the glue logic to get the captured image where you want it. Do you have an SD card shield for the board? (i.e. https://store.arduino.cc/usa/mkr-sd-proto-shield) It's probably best to interface with that via the SAMD21. You could use the FPGA<->SAMD21 SPI interface to stream a captured image from the SDRAM onto the SD card. That would require some work around changing the FIFO receiver to output over SPI instead of to the HDMI interface. Or you could have a 3rd, lower priority tenant in the arbiter so you still get a live preview on the screen.

Additionally, I am unfamiliar with HDLmake, is this tool also necessary for compiling?

It's not required, more a thing of personal preference. I'm using it to simplify my dev cycle (create project, testing, synthesis) so I can code in VSCode and usually don't need the Quartus GUI. But you could just as well create your own project and add the source there.

[benseleb]

Thanks @sameer,

Thanks for the reply!

It's not required, more a thing of personal preference. I'm using it to simplify my dev cycle (create project, testing, synthesis) so I can code in VSCode and usually don't need the Quartus GUI

If that's the case, I'll try creating implementing the source into my own Quartus Project. Does your project use it's own pin mapping, or the original one supplied by the Arduino Devs? I noticed names are slightly different.

You could use the FPGA<->SAMD21 SPI interface to stream a captured image from the SDRAM onto the SD card. That would require some work around changing the FIFO receiver to output over SPI instead of to the HDMI interface.

I do have an SD breakout for the board (MKR ENV SHIELD), and your previous recommendation is the method I intend on using. Also, I'm curious if it is as simple as changing the I2C command to the camera to swap between capturing a stream of frames to just a single frame.

Again, sorry if I'm butchering jargon but I am new to the world of FPGAs. I really appreciate your help!

[sameer]

I noticed names are slightly different.

The physical pins are the same, it's more a matter of naming preference. You should be fine using either with some minor renaming

as changing the I2C command to the camera to swap between capturing a stream of frames to just a single frame.

You could definitely do something like that.

The way these MIPI camera modules (and cameras in general maybe, but not 100% sure) work is that they are continuously streaming images. When you take a picture on a modern phone it'll stack multiple frames to remove artifacts and improve visual quality (i.e. night vision on Android for low light pictures).

But if your goal is to get a single frame onto the SD card, you could just start the camera, capture the frame, and stop the camera. (forgive my really bad drawing skills, meant to digitize this, but here's the camera control state diagram)

Again, sorry if I'm butchering jargon but I am new to the world of FPGAs. I really appreciate your help!

Yeah no worries, I'm also a newbie 😅

[benseleb]

Okay, great! I'm gonna go ahead and get my hands dirty then and see what I can get working.
Again, thanks for your help! I hope you don't mind me coming back to ask questions every now and then.

[benseleb]

Hi Sameer,

I've been able to assemble most of your work in the Quartus IDE, but am now experiencing some trouble with the pin mapping.

In particular, I seem to be running into issues with the differential i/o pins that exist for the MIPI and HDMI connections. From all the Vidor pin mapping scripts I've seen (like yours below), MIPI and HDMI clocks and transmit pins require LVDS format.

After formatting, here is what some of my LVDS pin mapping for the HDMI pins looks like:

Now when I compile using Quartus, I get multiple errors citing that these differential pins are located too close to other non-differential pins (I'm aware this could lead to interference).

Unless my setup of the LVDS pins was wrong, or using LVDS was incorrect, I'm not sure how to avoid this issue. The pin locations are pre-defined by the board layout, so I imagine I must be missing something. I've read that as long as the nearby non-differential pins aren't switching often, I can apply an I/O Maximum Toggle Rate assignment of 0MHz on those single-ended pins to bypass this error message. --However, I don't know how often the HDMI transmit pins are switching, so I figured I'd ask you for some insight.

Have you run into this situation before? Or can you tell if I have done anything wrong in my pin assignments? Any help would be appreciated!

[sameer]

That error is new to me, what version of Quartus are you using? Also could you share the full qsf? Wondering what constraints are different between yours and mine.

Tried synthesizing on my end and it's working with Quartus 20.1.1 (there was one issue that I've fixed in 4808bfb).

[benseleb]

Huh, that's weird. I'm using Quartus 20.2.1 as well.

Here is the modified qsf I tried using to import the pin assignments. It's a combination of your pin mapping and the original Arduino github mapping.
vidor_s_pins_collide.txt

It correctly named the pins and locations, but did not assign their format as it should -- (all just got set to the default voltage). Because it wouldn't import the formats, I ended up assigning everything manually.

I've exported my current pin assignments to the following file:
mipi_demo_mapping.txt

[benseleb]

I've since looked at some of the official Vidor projects -- like VidorGraphics, and I saw that many of the pins that were assigned to LVDS in the .qsf file were set to a regular 3.3V within Quartus pin mapper. Only the clock pins remained in LVDS format.
Not sure if this was just an error in implementation, or if LVDS is unnecessary for many of the pins.