I want to thank all of the people who commented on my first post about this project, I can’t believe how much time has passed since I first wrote about this. I actually did get it running but ran out of time to document it all while getting it working… I planned show it at a conference for work and ran into significant hurdles along the way so my documentation slipped a bit. I’m going to finish this out thanks to the extra motivation I received from everyones interest. So let’s get to it!
- Setting up a Hyperledger Fabric development environment on a Raspberry Pi
- Building Hyperledger Fabric on Raspberry Pi
- Setting up a Docker Swarm on Raspberry Pi
- Deploying a Hyperledger Fabric network on the Swarm with Docker Stack and testing with BYFN.
Mastering the build step is a pretty decent challenge. In this article I’ll pass along some tips and hints on how I pulled it off.
TLDR
Most of the content in this section is dedicated to get a working build on a Raspberry Pi. If you have no interest in doing a build yourself you can skip it and use my images. They aren’t part of the official hyperledger docker hub image set but I’ve got them out on mine for connivence. You can pull them as follows:
docker pull jmotacek/fabric-baseos:armv7l-0.3.2
docker pull jmotacek/fabric-basejvm:armv7l-0.3.2
docker pull jmotacek/fabric-baseimage:armv7l-0.3.2
docker pull jmotacek/fabric-ccenv:armv7l-1.0.7
docker pull jmotacek/fabric-javaenv:armv7l-1.0.7
docker pull jmotacek/fabric-peer:armv7l-1.0.7
docker pull jmotacek/fabric-orderer:armv7l-1.0.7
docker pull jmotacek/fabric-buildenv:armv7l-1.0.7
docker pull jmotacek/fabric-testenv:armv7l-1.0.7
docker pull jmotacek/fabric-zookeeper:armv7l-1.0.7
docker pull jmotacek/fabric-kafka:armv7l-1.0.7
docker pull jmotacek/fabric-couchdb:armv7l-1.0.7
docker pull jmotacek/fabric-tools:armv7l-1.0.7
Doing a build yourself…
First off don’t do your builds over ssh. It’s takes a while and if your tunnel is interrupted the process will fail (unless you execute the build with a special command but I won’t get into that…).
Clone the Repos
We need to get both the project and base image sources built in order.
mkdir -p go/src/github.com/hyperledger
cd $GOPATH/src/github.com/hyperledger
git clone -b armv7l-0.3.2 https://github.com/Cleanshooter/fabric-baseimage.git
git clone -b release-1.0 https://github.com/hyperledger/fabric.git
I submitted a pull request to the hyperledger team because right now their base image build targets the latest version of the armv7/armhf-ubuntu docker image. However, at the time of this post, the build won’t work unless you use xenial. My repo above is a mirror of the original with the slight change to the Makefile to ensure we target the correct image. In the meantime please feel free to use my clone of their repo, I’ll try and remember to update this post if they accept the pull request (leave me a comment if you notice they accepted the pull before I do).
You might also notice that branch targets. Currently the HyperLedger team is working on V1.1 and this tutorial is for V1.0. So We need to target specific version of the repos to ensure compatibility.
Also, the official documentation says to clone from the Gerrit repo… which requires you to sign up for an LFID… which is a pain in the ass. It’s ONLY NECESSARY if you actually plan to contribute code. I’m not a Go programmer, nor do I want to be… so cloning from the Gerrit repo is unnecessary. The github repo is a “read-only” copy of the Gerrit repo and says it is updated after every commit they make. However, I’ve run into several situations where they have pull requests in gerrit or say DONE in JIRA that the changes are not reflected in the git version.
I guess you also have to sign up for an LFID if you want to join their chat. Personally the chat was hit or miss for me. In my experience 60-80% of any questions you put to the fabric channel will go unanswered. The channel is constantly littered by people asking for help setting up a non “BYFN” networks, posting error logs and only getting a response half the time, and asking for answers to questions that honestly could be found if they looked through the documentation provided. It doesn’t help that the current setup documentation really only covers one super simple use case (the BYFN). At this point in the project I think the expectation is that you will just have to figure out everything beyond the basics yourself, which means there is a really steep learning curve. Most of the time if you ask a question you just won’t get an answer. To be fair they really should have a moderator for the channel telling people to post their questions out on StackOverflow because the same questions are being asked over and over (which is probably why they are being ignored). Granted there are a few people on there that will help others when they know the answer or are online. Chat channels are nice but when a project of this complexity is open to the public and there’s little support for it… it kinda leaves you feeling more frustrated than supported.
Six months later after writing the above… It seems they are starting to use StackOverflow more which is great. I haven’t been on the rocket chat in forever but I might have to since I’m finishing this documentation.
Build the Base Images
PLEASE NOTE: The latest version (0.4.7 as of this post) of the baseimages DO NOT WORK WITH HL v1.0 anymore, if you attempt to build HL v1.0 with the latest baseimage it will fail, my experience lead me to this issue on their JIRA (which is unresolved, granted it’s just a linter failure but it looks like the GOPATH isn’t set correctly in the baseimage at the moment so I’d avoid it for safeties sake).
I did build out the v0.3.2 ones after fixing an updating some things in my own clone of the repo. This will create the amrv7l baseimages on v0.3.2 as local docker images.
cd fabric-baseimage
make docker
Building Fabric
Well grab your loom and prepare yourself for some serious waiting. Lets weave this fabric!
Now that we have our ARM base images and our repo cloned we can build hyperledger fabric itself. Navigate to the ~/go/src/github.com/hyperledger/fabric/ directory, which if you built your own base images is right next to it.
If you pulled the base images from my docker hub set the BASE_DOCKER_NS, ARCH and BASEIMAGE_RELEASE as I have below, otherwise remove it. If you built your own base images you can get rid of the variables.
make native docker license spelling linter unit-test BASE_DOCKER_NS=jmotacek ARCH=armv7l BASEIMAGE_RELEASE=0.3.2
Build Notes
The makefiles “dist-clean” command will do nothing in this environment since the project doesn’t officially support armv7l so nothing will get cleaned up. So if your build fails you’ll have to delete the release/linux-armv7l/hyperledger-fabric-linux-armv7l.$(PROJECT_VERSION).tar.gz yourself to rebuild. Personally I’ve been wiping the entire fabric directory and re-cloning the repo just to be safe.
Also, behave tests no longer work in v1.0. I suspect it’s because the build scripts pull tests directly from their repo (the latest repo which means it includes v1.1) and the new tests rely on libraries that aren’t part of the v1.0 builds (like the b3j0f.aop pip library). You may be able to add these to your development environment but the behave tests aren’t necessary, see the documentation as to why.
Builds Complete!!!!
It’s always a hurdle to debug builds and get them working so I’m glad I’ve got some to start testing out. Our next step is setting up a docker swarm across a few of my raspberry pi’s. This step is pretty easy compared to this one so I’m hoping to have the next article in this series out soon.
If you run into any issues let me know and post a comment.
Hi.
Completed yesterday the instruction, and so far all good. Still to test a proper network.
Good job!
Just a suggestion though: the instructions aren’t very well structured, when compared with part 1.
I started inputing commands, and only after reading that they were not necessary, and could just copy your image. Inputed unnecessary commands a lot of times. Consider changing the instructions structure, making more clear the alternative instalation steps.
Planning to upgrade this to fabric 1.1?
Cheers
Thanks for the feedback Pedro. I’ll add in a TLDR section at the top so people can skip this step if they are just gonna use my images. Part of the reason I started documenting this to put my experiences in building it somewhere so that way when I go to do the v1.1 build I have notes somewhere on how I did it and things I had to do to get this build working. So unless you are trying to build the images most of the step’s content aren’t super relevant.
Hi Joe,
great instructions so far, thank you! Unfortunately I run into some trouble when compiling, which is the unit-tests fail. Now, does that matter (as you write behave Tests don’t)? Then also I don’t find anything inside the release folder, which probably is a consequence of the unit-test fail…
All the best
Hmmm… the unit tests passed on my build. The release folder isn’t the goal of the build, its the docker images that we are looking for. What do you see when you run the ‘docker images’ command? Did all the images I have in my list populate on yours? If not we’ll probably need to start troubleshooting your build config and ensuring your versions are aligned.
Hello,
I have trouble running executing the “make …. unit-test” command.(Getting the fabric compiled)
The error is thrown at the Makefile line 263: recipe for target — ‘build/…./protoc-gen-go’: No such file or directory
Hello,
The build recipes were working fine for — native, docker, then the license fails asking to update the committed files with SPDK … apache 2.0 .
May i know what exactly is happening here, or this make recipe can be ignored and considered as to have the build completed with native and docker recipes.
Thank you
The entire build completed as per the instructions.
There was a hurdle in making the ./byfn.sh -m up work to run the first network on PI.
The error is same as the issue posted on Cleanshooter’s repo – hyperledger-pi-compose.
I would like to know if there is any way to resolve the contingent error on the Chaincode step to complete the process.
Thank you
Raju
I was planning to write up the BYFN stuff today and since you found my repo already I’ll have an updated and working version there sometime soon.
Not sure what’s happening for you… could you post error logs?
Yeah so did I… and I ran into the same problem… can’t remember what I did to resolve it at the moment. Do you ever get through it?
Hey Joe,
Following your instructions and altering them where appropriate I have managed to get quite far at building Hyperledger Fabric for the arm64v8 architecture.
The base images (After heavy alterations) managed to build successfully but while building the Hyperledger Fabric images I am stuck at the couchdb image.
Specifically, I edited the Dockerfile which creates that image and added the lines “node -v” and “npm -v” (As npm install -g grunt-cli was failing) and it throws an Invalid Instruction.
I have even tried downloading a fresh pre-compiled arm64v8 Node binary and it still throws that error. Any thoughts?
Thanks in advance and keep up the good work, I am looking forward to your BYFN tutorial.
Wow I’m really interested in your use case. Could you point me to your repo (with edited Dockerfiles)? I’d like to see the error message from your CouchDB build as well… can you share? I’ve had to try and figure out a multitude of build issues throughout my adventures with HL so I might be able to help with some more context… hells I might try to do an ARMv8 build myself with HL v1.1 after I finish the BYFN tut.
Hey Joe,
Thanks for your response and sorry for the late reply! I actually managed to fix all issues I faced and built HL v1.0.6 for ARMv8.
Seeing a v1.1 build would be interesting as I wasn’t able to bypass the issues I faced there.
Building Node from source actually fixed the issue since the post-make script installed quite a few stuff so I suspect a dependency was missing.
I am sure you faced this as well but most apache links are broken in HL v1.0 so you have to use the alternative archive.apache ones for CouchDB etc.
I was directly editing the generated Dockerfiles within the build directory as I am not sure where they were derived from, do you have any idea where they are located so I can alter them accordingly and create a proper Github fork?
The unit-tests also failed for me but I have successfully booted the chaincode-docker-devmode network and interacted with custom chaincode on it. All images were also successfully created.
My use case is utilizing blockchain-based authentication on IoT environments, it is actually a commercial project.
Thanks in advance.
Congratulations on troubleshooting the build!
I’m actually going to start working on a v1.1 build myself this month.
I did face the same problem with the apache links, glad you were able to trouble shoot them. I actually used to have a fork with the updated URLs in it but after I submitted a pull request, they updated it themselves so I deleted it. The pre-build files where the URLs are located are in the images folder then depending on which one is outdated you update that one respectively.
I’m super jealous that you are on a commercial project… and your use case sounds like a great fit for the tech. Good luck!
Hi,
i’m really thankful for your tutorial and it is pretty helpful. I’ve already run part 2 successfully on my first pi. But my second pi leaves me with an error when trying to execute
make native docker license spelling linter unit-test BASE_DOCKER_NS=jmotacek ARCH=armv7l BASEIMAGE_RELEASE=0.3.2
.
I’m running to the error
The command ‘/bin/sh -c set -x && cd / && curl -fsSL “http://www.apache.org/dist/zookeeper/$DISTRO_NAME/$DISTRO_NAME.tar.gz” | tar -xz && mv “$DISTRO_NAME/conf/”* “$ZOO_CONF_DIR”‘ returned a non-zero code: 2
Makefile:282: recipe for target ‘build/image/zookeeper/.dummy-armv7l-1.0.7-snapshot-1575ea62’ failed
make: *** [build/image/zookeeper/.dummy-armv7l-1.0.7-snapshot-1575ea62] Error 2
.
Maybe you can help me with this problem. I would be really greatful for.
Thank you in advance,
Jenna
Hey Jenna glad it’s been helpful. First off you only need to run this step on one RasPi. Once you have the images built for the version you want to work with you can simply push them to a repo like Docker hub where the other RasPis can pull them from.
To the error message specifically:
When I ran into this error in the past it was due to resource location changes. Zookeeper people moved their project files to another location and no one has updated the release-1.0 repo yet to point to the new location (probably an archive). I’ve noticed that this is actually pretty common for apache teams/projects to do and IMHO super annoying. I think they do it on purpose so that projects using their software stay up to date with the latest and greatest version. So this error isn’t your fault. There are two way to fix it…
1. Wait for the HL team to update their repo to the new location. You can put in a PR or log an issue with them if you’d like.
2. Fork the release-1.0 source yourself, find the current URL for the version of zookeeper we need and replace the variables in the make script yourself. Then pull your repo instead of the one I’ve listed above.
Hi,
I’m trying to build fabric v1.1.0 on Pi, but I faced a error during compile procees.
pi@esir:~/go/src/github.com/hyperledger/fabric $ make peer-docker
Building build/docker/bin/peer
# github.com/hyperledger/fabric/vendor/github.com/milagro-crypto/amcl/version3/go/amcl/FP256BN
vendor/github.com/milagro-crypto/amcl/version3/go/amcl/FP256BN/FP.go:103:16: constant 30564559323915749 overflows int
Makefile:227: recipe for target ‘build/docker/bin/peer’ failed
make: *** [build/docker/bin/peer] Error 2
Can you help me fix this issue?
Thank you in advance.
This is a tough one… since you are building one image specifically. Honestly I’d have to review the go code in the repo where the int is overflowing, find the variable name, see if it’s elevated to the config where you could set it via the dockerfile during the build. It’d be a lot of digging but that would be my suggestion on where to start.
Hyperledger does not support 32-bit architecture yet –> https://jira.hyperledger.org/browse/FAB-10454?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
It looks like Jiahua answered his earlier question. Thanks for posting a link to the answer… I was about to start building some v1.1 images myself and this will save me a few hours, cheers!
Joe,
I got all the way through the pull steps and everything worked fine. Then, I started to build the ledger fabric and received this message –
make: *** No rule to make target ‘native’. Stop.
I have no idea what to do now. I googled and found nothing relevant to this situation. I have no idea where to find the makefile so I cannot even debug.
Any ideas?
Thanks in advance,
Ken
Thank you so much Joe!
I have successfully made images of Hyperledger Fabric 1.2.1 baseimage 0.4.11 if anyone is interested.
There build ontop of pi64 from Bamarni and tested the fabric-samples fabcar from the tutorial.
docker pull nicoswan/fabric-baseos:arm64-v1.2_v0.4.11
docker pull nicoswan/fabric-basejvm:arm64-v1.2_v0.4.11
docker pull nicoswan/fabric-baseimage:arm64-v1.2_v0.4.11
docker pull nicoswan/fabric-ccenv:arm64-v1.2.1_v0.4.11
docker pull nicoswan/fabric-javaenv:arm64-v1.2.1_v0.4.11
docker pull nicoswan/fabric-peer:arm64-v1.2.1_v0.4.11
docker pull nicoswan/fabric-orderer:arm64-v1.2.1_v0.4.11
docker pull nicoswan/fabric-buildenv:arm64-v1.2.1_v0.4.11
docker pull nicoswan/fabric-testenv:arm64-v1.2.1_v0.4.11
docker pull nicoswan/fabric-zookeeper:arm64-v1.2.1_v0.4.11
docker pull nicoswan/fabric-kafka:arm64-v1.2.1_v0.4.11
docker pull nicoswan/fabric-couchdb:arm64-v1.2.1_v0.4.11
docker pull nicoswan/fabric-tools:arm64-v1.2.1_v0.4.11
The build of the images, etc took about 3 days so hopefully this will save everyone some time.
Im still testing but it looks promising.
Thank you for all the guys commenting in the post as it really you and Joe that made it possible for me 🙂
Nico
could not access image
docker pull nicoswan/fabric-javaenv:arm64-v1.2.1_v0.4.11
I stumbled upon an issue here when trying to build on Raspberry Pi 3B+ with latest fabric-baseimage and HL Fabric 1.4 on Raspbian stretch lite.
1.4 requires GO 1.11.5, which apparently only outputs “ARM” as GOARCH. The build fails an claims “arm” is an unsupported architecture. for this to work, you need to replace l23 in the Makefile:
old: ARCH=$(shell go env GOARCH)
new: ARCH=armv7l
Yeah more recent versions of HLF have build issues on ARM as the underlying tech has gotten more developed and outpaced available ARM libraries.
hyperledger has stopped the nexus repository: https://lists.hyperledger.org/g/fabric/topic/hyperledger_nexus_repository/70049382?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,70049382
That results in a failing make:
> make native docker license spelling linter unit-test BASE_DOCKER_NS=jmotacek ARCH=armv7l BASEIMAGE_RELEASE=0.3.2
I ran into similar problems during the builds and it comes down to raising the issue on their Github/Rocket (whatever) repo for the version you are targeting. Basically they need to update the URL for the source repo. Likely that the source repo has been updated/moved/archived.
OR
Cloning the HL project and updating the URLs yourself. All depends on how invested you are into fixing the build issues.