Author Topic: [Cross-platform] FS2 GOG/Mod installer  (Read 81498 times)

0 Members and 1 Guest are viewing this topic.

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Updated "Ambiance" style sheet (combobox contrast fix, border fix, button height):
Code: [Select]
#toolbar {
    padding:3px;
    padding-bottom: 4px;
    max-height: 50px;
    background: qlineargradient(spread:pad, x1:0, y1:0, x2:0, y2:1, stop:0 rgba(63, 62, 58, 255), stop:1 rgba(88, 86, 78, 255));
}

#toolbar QPushButton {
    height: 20px;
    color: #DFDBD2;
    background-color: transparent;
    border-image: url(./button-toolbar.png) 8;
    border-top: 8px;
    border-bottom: 8px;
    border-right: 8px;
    border-left: 8px;
}

#toolbar QPushButton:hover {
    border-image: url(./button-toolbar-hover.png) 8;
}

#toolbar QPushButton:pressed {
    border-image: url(./button-toolbar-pressed.png) 8;
}

#toolbar QPushButton:focus {
    border-image: url(./button-toolbar-focused.png) 8;
}

#toolbar QLineEdit {
    height: 20px;
    background-color: transparent;
    border-image: url(./entry-toolbar.png) 8;
    border-top: 8px transparent;
    border-bottom: 8px transparent;
    border-right: 8px transparent;
    border-left: 8px transparent;
    color: grey;
}

#toolbar QLineEdit:focus {
    color: #4C4C4C;
    border-image: url(./entry-toolbar-focused.png) 8;
}

#toolbar QComboBox {
    color: #DFDBD2;
    height: 20px;
    padding: 0px;
    min-width: 6em;
    background-color: transparent;
    border-image: url(./button-toolbar.png) 8;
    border-top: 8px;
    border-bottom: 8px;
    border-right: 8px;
    border-left: 8px;
}

#toolbar QComboBox:hover {
    border-image: url(./button-toolbar-hover.png) 8;
}

#toolbar QComboBox:focus {
    border-image: url(./button-toolbar-focused.png) 8;
}


#toolbar QComboBox::drop-down {
    border: none;
}

#toolbar QComboBox::down-arrow {
    image: url(./combobox-arrow.png);
    subcontrol-origin: padding;
    subcontrol-position: top right;
    padding-left: 4px;
    width: 20px;
    height: 20px;
    border-left-width: 1px;
    border-left-color: #353531;
    border-left-style: solid;
    border-top-right-radius: 3px;
    border-bottom-right-radius: 3px;
}


#toolbar QComboBox QAbstractItemView {
    background-color: #353531;
}

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
This will hopefully be the last delay in the team update. I ditched most of the previous team logic, and I'm rebuilding it nearly from scratch.

I have to manage the user-team relation, and there were 3 options :
- hardcode team permissions (admin, member, invited modders...)
- use standard Symfony ACL
- do something else (some kind of custom ACL+ relation attributes)

I experimented with the first one (hardcoded permissions), but it's not flexible enough. It doesn't really cause any issue yet, but maintainability would end up being a problem.
The second one (Symfony ACL) would be ok, except I still may have to store relation attributes that are not permission related (i.e. : join date). Managing the user-team relation with 2 completely different Symfony methods (ACL+Doctrine2 mapping) feels like a bad idea in the end.

That's why I'm going to create a small class that works a bit liks ACL (with lexx complexity), but can also store any other type of data about user-team relation. It should be good enough for now, and shouldn't break if it's extended.

  

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
@m!m: I've changed the mod.json documentation. Is this better?
I'll add a "filesize" property to mod archives. That way you can determine the download size without using any HTTP requests. What do you think?

@Hellzed: I'm curious how your permission system works / looks in the web interface once it's finished. If I have enough time, I'll try to implement detection of obsolete links and I'll make the Nebula delete old files from the mirror.



I've finally finished the packages for Knossos. It runs pretty well on Windows, Linux (tested on Arch, Ubuntu and Xubuntu) and Mac OS X.



On Windows and Ubuntu the current download progress is displayed in the Taskbar.

Installation instructions

If you encounter any problems or bugs, you can post here or create a new issue on GitHub.

EDIT: Updated.
« Last Edit: February 05, 2015, 07:04:15 am by ngld »

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
Awesome !

Permissions are managed through a bitmask system associated with human-readable "roles", and these roles will be displayed in a html "select" element for each member of the team (editable by team admin).
I'm still figuring some implementation choices since this is not a standard many-to-many Doctrine2 ORM relation, and if i do something wrong it will drown the SQL server under queries (well, there's not that much data, but 50 queries to generate a 10 member team page would be awful). It also makes the code to access a team's members and a member's teams much more complicated, so I'll need to write some helper functions.
In the end, the added code will not be that long (even quite short), but it's a matter of doing everything right.

 

Offline niffiwan

  • 211
  • Eluder Class
Re: [Cross-platform] FS2 GOG/Mod installer
Very cool  :yes:

I have just one question at this point; how do you select a new FSO dir? I originally selected my "real" FSO dir, but now I want to switch to my test FSO dir before I try installing any mods. Just in case...

ps. I tried the pip install to start with; I think you need the --pre flag in order for this to work correctly.  And maybe other stuff as well; but I didn't troubleshoot for too long before just using the PPA instead.
Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Click on "Settings", select "Game settings" on the left and click the "Browse" button on the right.

 

Offline niffiwan

  • 211
  • Eluder Class
Re: [Cross-platform] FS2 GOG/Mod installer
Thanks for that - I tried that and was confused because the dropdown still listed the exec's from the other directory (and my test dir didn't have any execs in it at that point which may have contributed to the issue  :nervous:)
Creating a fs2_open.log | Red Alert Bug = Hex Edit | MediaVPs 2014: Bigger HUD gauges | 32bit libs for 64bit Ubuntu
----
Debian Packages (testing/unstable): Freespace2 | wxLauncher
----
m|m: I think I'm suffering from Stockholm syndrome. Bmpman is starting to make sense and it's actually written reasonably well...

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Oh, that's a bug. I forgot to update the dropdown if you change your FS2 path.

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
@m!m: I've changed the mod.json documentation. Is this better?
Yes that's better, thank you!

I'll add a "filesize" property to mod archives. That way you can determine the download size without using any HTTP requests. What do you think?
I think that's a great idea! With that the UI could display an estimated download size and maybe an estimated completion time without the added overhead of sending HTTP requests to all URLs.

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
@ngld:
I kept reading advanced doc for Symfony. Going back to the drawing board with the whole Nebula security implementation (again, but this time not juste teams. The whole security !).

I really have to stop thinking of Nebula as a "small" project. I have made some implementation choices (security, but not only that part, also some database management) because I read some things (namely Symfony default ACL, and Symfony paginator) would be overkill for a "small" project. Now I figure it's not, because we actually need features like administration granularity...
Our talk about Nebula admin helped a lot ! :)

1) I'm going to drop our custom security voters (I always planned to rewrite this part anyway, because it lead to some terrible variable naming in some of the html templates), replace it with standard ACL, and enable an early version of admin access to mods. Creating the admin panel will be much easier once it's done.
2) Going back to http://www.hard-light.net/forums/index.php?topic=86364.msg1768398#msg1768398 , I'm keeping the special "attributes" class I was writing to store data about a relation between a modder and a team, but instead of writing a custom ACL, I will use the standard ACL system so Nebula will have a single big authorization system.
(at first, I thought this would add complexity, but when I drop the voters, it will get much more simple)

Dropping custom security voter classes is also important because Symfony devs will make deep changes starting with Sf 2.6 that may completely break our security system. ACL won't change any time soon.

Less custom, more standard code will also make my code more readable to any other Symfony dev, in case someone wants to contribute.

EDIT : Good news ! It's working (already rewritten the "mod owning" part... Still have to do the branches and builds...).
Bad news : database reset coming (we can keep mods created with the old voter system, but their owners will be unable to make changes).

EDIT 2 : Since I'm changing a lot of things in the db, I'll simplify a few more things to play better with the new security system:
- No more FSMod, Team objects ! Team + Owner interface (the "repository") + FSMod will become a unified object, some kind of "work group".
- Users will own the work groups in a standard many-to-many relation, permissions will be managed through the ACL system.
- Aggregated fields (mod, branch, build counters) will be handled with more efficiency through the work groups.

---> For users, it means they won't have to worry about team management : they create a mod, and if they need to give rights on this mod to other users, they can do it through the mod admin page. Every mod will retain a "team" page, showing the current authorized members.
---> For me, that means : better top level templates and better top level controllers (no need to check if a mod is owned by a user or by a team), less duplicated code (no need to handle mod owning in team objects), easier checks when an object (user, mod) is created...
It's actually close to making teamwork a "default" on Nebula.

These changes will have no impact on the lower structure (build content).

EDIT 3 : This refactoring will go even deeper, with a better build validator abstraction !
Currently, I use a service to query knossos-server api from Controllers. The Controllers then 1) read the answer, 2) compare with the database, and 3) take appropriate action.
After the new changes, the "read the answer" and "compare with the database" steps will be included in the service, so the Controllers will only need to send the right page depending on the aggregated answer.
This will lower the risk of redirection bugs in Nebula build validation pages (in other words, I will be able to understand my own redirection code, which is currently barely the case, because it's a nested conditions hell).
« Last Edit: November 19, 2014, 01:18:41 pm by Hellzed »

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I've updated knossos-server (the converter part of the Nebula). It can now queue build requests if all workers are busy and you can watch running tasks.

Knossos also received an update. I've (hopefully) fixed all the known bugs and added a new feature which allows the user to install and uninstall packages for installed mods.
Windows and Mac users should reinstall using the links in the release post. Linux users can user their package manager or if they installed knossos with "pip install", they can use "pip install -U --pre knossos".

 

Offline Hellzed

  • 28
Re: [Cross-platform] FS2 GOG/Mod installer
I spent the last week getting ready to relocate (new city, new country, new job)... the big refactoring is still going on though, and I'm currently waiting for an answer from Symfony devs about one of the built-in components.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Well, I guess I've implemented most of the important features. Once my current TODO list is finished, I'll release Knossos 0.1.0.

Recent Changes:
  • Fix issues on Linux where FSO depended on missing libraries
  • Improved the build system
  • Refactored repo storage (mods are now stored in ~/.knossos/mods.json or %APPDATA%\Knossos\mods.json)
  • The converter can now download files from box.com
  • Added (experimental) version selection for mods.

@m!m: I added a new "executables" property to packages. It's not documented, yet, because I wanted your feedback on it. It allows a package to list all included executables. FSO's Windows package would have something like this:
Code: [Select]
"executables": [
  {
    "name": "3.7.2 RC4",
    "file": "fs2_open_3_7_2_RC4.exe"
  },
  {
    "name": "3.7.2 RC4 (Debug)",
    "file": "fs2_open_3_7_2_RC4_DEBUG.exe"
  }
]
This way it's easier to put together a list of FS2 Open executables and it gives the packager the opportunity to specify nice labels. What do you think?

I've also added a new type to the packages' "environment" check. It's called "bool" and allows the packager to write down an expression like "(macos || linux) && X86_64 && sse".
@m!m: The converter automatically parses that expression. The above expression would appear in the repo like this:
Code: [Select]
"environment": [
  {
    "type": "bool",
    "value": ["and", ["and", ["or", ["var", "macos"], ["var", "linux"]], ["var", "X86_64"]], ["var", "sse"]]
  }
]
If you're interested, I can give you the full list of available variables.

Things still left TODO:
  • UI changes
    • Display current progress in a "progress" tab. Right now Knossos opens a new window for this.
    • Replace the combo box with flat buttons.
  • Make sure I've fixed all bugs in the settings window. (Mostly related to saving and updating the shown values.)
  • Check the version selection for mods and make sure it works correctly.

Things I plan to do after I release Knossos 0.1.0:
  • Extend the mod installation dialog.
  • Mac OS X: Support for innoextract
  • Mac OS X: Display icons in the WebView correctly. This is most likely a bug in Qt's WebKit implementation.

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
@m!m: I added a new "executables" property to packages. It's not documented, yet, because I wanted your feedback on it. It allows a package to list all included executables. FSO's Windows package would have something like this:
Code: [Select]
"executables": [
  {
    "name": "3.7.2 RC4",
    "file": "fs2_open_3_7_2_RC4.exe"
  },
  {
    "name": "3.7.2 RC4 (Debug)",
    "file": "fs2_open_3_7_2_RC4_DEBUG.exe"
  }
]
This way it's easier to put together a list of FS2 Open executables and it gives the packager the opportunity to specify nice labels. What do you think?
I like that but it would be even better to replace "name" with something like this:
Code: [Select]
  "version": {
    "major": 3,
    "minor": 7,
    "release": 2,
    "debug": true,
    "build": 42,
    "tags": [
      "RC4"
    ]
  }
The launcher can then decide how to display that executable.

I've also added a new type to the packages' "environment" check. It's called "bool" and allows the packager to write down an expression like "(macos || linux) && X86_64 && sse".
@m!m: The converter automatically parses that expression. The above expression would appear in the repo like this:
Code: [Select]
"environment": [
  {
    "type": "bool",
    "value": ["and", ["and", ["or", ["var", "macos"], ["var", "linux"]], ["var", "X86_64"]], ["var", "sse"]]
  }
]
If you're interested, I can give you the full list of available variables.
TBH I don't think we need this level of fine control for environment flags. It is only needed for executables for which our existing system should be sufficient.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
I like that but it would be even better to replace "name" with something like this:
Code: [Select]
  "version": {
    "major": 3,
    "minor": 7,
    "release": 2,
    "debug": true,
    "build": 42,
    "tags": [
      "RC4"
    ]
  }
The launcher can then decide how to display that executable.

Can we agree on the semver standard and shorten it to something like this?
Code: [Select]
  "version": "3.7.2-RC4+42",
  "debug": true

 

Offline m!m

  • 211
Re: [Cross-platform] FS2 GOG/Mod installer
Yes, semver is much shorter and it contains all the necessary information.

 
Re: [Cross-platform] FS2 GOG/Mod installer
Just submitted a bug report on this tool. I cannot make it work. See report at https://github.com/ngld/knossos/issues/15
There are only 10 kinds of people in the world;
those who understand binary and those who don't.

 

Offline ngld

  • Administrator
  • 29
  • Knossos dev
Re: [Cross-platform] FS2 GOG/Mod installer
Oh, look an update!  :nervous:

The nebula now has its own domain! You can reach it at https://fsnebula.org/nebula. The complete site is served over SSL.
However, until Hellzed publishes the next version, it won't get any new features.

Recent changes:
  • Mac OS X is now fully supported! (innoextract works and I've fixed the icon issue.)
  • The progress dialog has been integrated into the main window.
  • The select box which allowed you to select whether you want to see installed/available mods or updates has been replaced with a bunch of buttons. This should make navigating faster.
  • The mod settings window now also displays the mod's description and logo. It also has a new "Troubleshoot" tab. There you can check for corrupted, loose and missing files and replace them.
  • The mod install logic has been improved:
    • The launcher now shuffles the mirror list to distribute the load across them.
    • Downloads are retried up to 3 times per mirror.
    • The checksum of each downloaded file is checked to make sure that the download isn't corrupted.
    • I've fixed an issue on Windows where 7z couldn't extract an archive because it was still locked by Knossos.
  • The "Game settings" page (where you can change your FS2 path and select a FSO build) now also has a button to open the fs2_open.log file. This should make it easier to find the log.
  • FSO is now installed in a subdirectory by default, this allows the launcher to treat it just like the other mods. This works even on Windows. However, you won't be able to launch FSO without the launcher.
  • Knossos can read mod.ini files which means that it will be able to launch your installed mods. Mods which only have a mod.ini file (and no mod.json file) are displayed in the list with a "(ini)" suffix behind the mod title. These mods can't be updated nor repaired by Knossos since the necessary info (normally contained in the mod.json file) is missing.
  • I've fixed a few crashes related to PyQt4 and PySide.
  • I've alsof fixed a bug on Mac OS X related to OpenSSL.
  • I've set up a page with proper installation instructions. I'll link to it once I've finished the release.

I still have to fix two bugs:
  • After the user installs a mod, internal references to that mod are screwed up. Which might cause weird behaviour.
  • If the user hasn't used FSO before, he/she has to open the settings window and save before launching FSO.

I'll try to finish the 0.1.0 release this week.
The test repository contains a few mods and FSO 3.7.2 RC4. Once the release is out, I'll add RC5, Nightlies and Swifty's "Deferred Lighting and soft shadows" build. If anyone wants a specific mod added, just post a link to the mod's release post or download page and I'll try to add it ASAP.

@m!m: The link is https://fsnebula.org/repo/test.json. Your launcher has to support the "includes" field and TLS >= 1.0 to access that repo.

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: [Cross-platform] FS2 GOG/Mod installer
The amount of work put into this project in such a short time span astounds me.  I'm watching this project closely.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 
Re: [Cross-platform] FS2 GOG/Mod installer
Indeed. This is impressive.